blog by software engineer
https://www.joelonsoftware.com/
Joel on Software
2022-12-19
Joel Spolsky
<div><article>
	<p>Since the 1990s, the web has been a publishing place for human-readable documents. Documents published on the web are in HTML. HTML has a little bit of… <a href="https://www.joelonsoftware.com/2022/12/19/progress-on-the-block-protocol/">Read more “Progress on the Block Protocol”</a></p>
</article><article>
	<p>You’ve probably seen web editors based on the idea of blocks. I’m typing this in WordPress, which has a little + button that brings up a long… <a href="https://www.joelonsoftware.com/2022/01/27/making-the-web-better-with-blocks/">Read more “Making the web better. With blocks!”</a></p>
</article><article>
	<p>The other day I was talking to a young developer working on a code base with tons of COM code, and I told him that even before… <a href="https://www.joelonsoftware.com/2021/06/02/kinda-a-big-announcement/">Read more “Kinda a big announcement”</a></p>
</article><article>
	<div>
	<p>Sometimes simulating complex systems is the best way to understand them. <a href="https://www.joelonsoftware.com/2020/06/18/hash-a-free-online-platform-for-modeling-the-world/">Read more “HASH: a free, online platform for modeling the world”</a></p>
	</div>
</article>||||I|||| Skip to content
 * View menu
 * View sidebar

 Joel on Software

 Your host

 I’m Joel Spolsky, a software developer in New York City. More about me.

 Search
 Read the archives in dead-tree format! Many of these articles have been collected into four books, available at your favorite bookstore. It’s an excellent way to read the site in the bath, or throw it at your boss.

 Careers

 Ready to level up? Stack Overflow Jobs is the job site that puts the needs of developers first. Whether you want to take control of your search or let employers discover you, we’re on a mission to help every developer find a job they love.

 Looking to hire smart programmers who get things done? Stack Overflow Talent is a fully-customized sourcing solution that helps you understand, reach, and attract developers on the platform they trust most. Find the right candidates for your jobs. Learn more.

 For my day job, I'm the co-founder and CEO of Stack Overflow, the largest online community for programmers to learn, share their knowledge, and level up. Each month, more than 40 million professional and aspiring programmers visit Stack Overflow to ask and answer questions and find better jobs. Stack Overflow is also the flagship site of the Stack Exchange network, 160+ question and answer sites dedicated to all kinds of topics from cooking to gaming. According to Quantcast, Stack Overflow is the 30th largest web property in the United States and in the top 100 in the world.




 I also founded Fog Creek Software, one of the most influential small tech companies in the world. As an independent, privately-owned company, we’ve been making customers happy since the turn of the century. We share what we've learned about how to make great software, both by writing about our ideas and by creating products, like FogBugz, Trello and Gomix, that help others make great technology. As a result, Fog Creek's impact on the world of developers rivals companies a thousand times our size.

 Twitter! Twitter!

 My Tweets

 What’s new?

 December 19, 2022 by Joel Spolsky

 Progress on the Block Protocol

 Since the 1990s, the web has been a publishing place for human-readable documents. Documents published on the web are in HTML. HTML has a little bit of… Read more “Progress on the Block Protocol”

 January 27, 2022February 1, 2022 by Joel Spolsky

 Making the web better. With blocks!

 You’ve probably seen web editors based on the idea of blocks. I’m typing this in WordPress, which has a little + button that brings up a long… Read more “Making the web better. With blocks!”

 June 2, 2021June 2, 2021 by Joel Spolsky

 Kinda a big announcement

 The other day I was talking to a young developer working on a code base with tons of COM code, and I told him that even before… Read more “Kinda a big announcement”

 June 18, 2020June 18, 2020 by Joel Spolsky

 HASH: a free, online platform for modeling the world

 Sometimes simulating complex systems is the best way to understand them. Read more “HASH: a free, online platform for modeling the world”

 Reading lists

 Over the last 16 years I’ve written 1114 articles on this site about software development, management, business, and the Internet. To make it easy to find the best ones, here are some reading lists, sorted by topic.

 Top 10

 * Things You Should Never Do, Part I
 * Strategy Letter I: Ben and Jerry’s vs. Amazon
 * The Joel Test: 12 Steps to Better Code
 * Fire And Motion
 * The Iceberg Secret, Revealed
 * The Law of Leaky Abstractions
 * The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
 * How Microsoft Lost the API War
 * The Perils of JavaSchools
 * The Development Abstraction Layer

 Stack Overflow

 * stackoverflow.com
 * Stack Overflow Launches
 * Stack Overflow is a Wiki
 * Stack Overflow Live!
 * Talk about StackOverflow
 * Server Fault now in public beta
 * Get a job
 * Upgrade your career
 * Programmer search engine
 * Stack stats
 * Raising money for StackOverflow
 * Facebook / LinkedIn importers
 * Stack Exchange 2.0
 * Sites for experts
 * News
 * Area 51 is now in beta
 * Web Applications Stack Exchange now in beta
 * A new WordPress Stack Exchange
 * Fork it!
 * New, PNG-based Stack Overflow flair
 * Stack Exchange for Jewish Life and Learning
 * The Wikipedia of Long Tail Programming Questions
 * Stack Overflow 2010 recap
 * Careers 2.0 (by Stack Overflow)
 * Stack Overflow (the company) is now Stack Exchange
 * Stack Exchange Raises $40m
 * The Stack Overflow Age
 * Announcing Stack Overflow for Teams
 * The next CEO of Stack Overflow

 New developer

 * NDAs and Contracts That You Should Never Sign
 * Getting Things Done When You’re Only a Grunt
 * The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
 * Getting Your Résumé Read
 * Mike Gunderloy’s Coder to Developer
 * Advice for Computer Science College Students
 * The Perils of JavaSchools
 * Talk at Yale: Part 1 of 3
 * Talk at Yale: Part 2 of 3
 * Talk at Yale: Part 3 of 3
 * Exploding Offer Season
 * The Duct Tape Programmer
 * Developers’ side projects

 Rock star developer

 * The Joel Test: 12 Steps to Better Code
 * Painless Bug Tracking
 * Daily Builds Are Your Friend
 * Don’t Let Architecture Astronauts Scare You
 * Hard-assed Bug Fixin’
 * A Hard Drill Makes an Easy Battle
 * Back to Basics
 * Five Worlds
 * The Law of Leaky Abstractions
 * Craftsmanship
 * Biculturalism
 * Foreword to Painless Project Management with FogBugz, by Mike Gunderloy
 * Making Wrong Code Look Wrong
 * Can Your Programming Language Do This?
 * Why are the Microsoft Office file formats so complicated? (And some workarounds)
 * Distributed Version Control is here to stay, baby

 Tech lead

 * Two Stories
 * Human Task Switches Considered Harmful
 * Lord Palmerston on Programming
 * The Development Abstraction Layer
 * Three Management Methods (Introduction)
 * The Command and Control Management Method
 * The Econ 101 Management Method
 * The Identity Management Method
 * Evidence Based Scheduling

 CEO

 * Incentive Pay Considered Harmful
 * Things You Should Never Do, Part I
 * Where do These People Get Their (Unoriginal) Ideas?
 * Top Five (Wrong) Reasons You Don’t Have Testers
 * Strategy Letter I: Ben and Jerry’s vs. Amazon
 * Strategy Letter II: Chicken and Egg Problems
 * Strategy Letter III: Let Me Go Back!
 * Wasting Money on Cats
 * Big Macs vs. The Naked Chef
 * Strategy Letter IV: Bloatware and the 80/20 Myth
 * Good Software Takes Ten Years. Get Used To it.
 * In Defense of Not-Invented-Here Syndrome
 * Fire And Motion
 * Strategy Letter V
 * Platforms
 * Rick Chapman is In Search of Stupidity
 * How Microsoft Lost the API War
 * Seven steps to remarkable customer service
 * Strategy Letter VI
 * Martian Headsets
 * Figuring out what your company is all about
 * Why testers?

 Startup founder

 * Fixing Venture Capital
 * Micro-ISV: From Vision to Reality
 * Foreword to “Eric Sink on the Business of Software”
 * My First BillG Review
 * Architecture astronauts take over
 * The eternal optimism of the Clear mind

 Software designer

 * Controlling Your Environment Makes You Happy
 * Figuring Out What They Expected
 * Choices
 * Affordances and Metaphors
 * Consistency and Other Hobgoblins
 * Designing for People Who Have Better Things To Do With Their Lives
 * Designing for People Who Have Better Things To Do With Their Lives, Part Two
 * Designing for People Who Have Better Things To Do With Their Lives, Part Three
 * The Process of Designing a Product
 * Painless Functional Specifications – Part 1: Why Bother?
 * Painless Functional Specifications – Part 2: What’s a Spec?
 * Painless Functional Specifications – Part 3: But… How?
 * Painless Functional Specifications – Part 4: Tips
 * Humane Programming
 * The Iceberg Secret, Revealed
 * Nothing is as Simple as it Seems
 * Building Communities with Software
 * It’s Not Just Usability
 * The Project Aardvark Spec
 * Introduction to Great Design (Second Draft, In Progress)
 * Great Design: What is Design? (First Draft)
 * What Makes It Great? (First Draft)
 * Amazing X-Ray Glasses from Sprint!
 * Choices = Headaches
 * Simplicity
 * Elegance
 * The Big Picture
 * Font smoothing, anti-aliasing, and sub-pixel rendering

 Product manager

 * How Many Lies Can You Find In One Direct Mail Piece?
 * Mouth Wide Shut
 * Camels and Rubber Duckies
 * How to demo software
 * Where there’s muck, there’s brass

 Recruiter

 * Whaddaya Mean, You Can’t Find Programmers?
 * Hitting the High Notes
 * Finding Great Developers
 * A Field Guide to Developers
 * Sorting Resumes
 * The Phone Screen
 * The Guerrilla Guide to Interviewing (version 3.0)
 * Programmer search engine

 Fog Creek Software

 There’s a software company in New York City dedicated to doing things the right way and proving that it can be done profitably and successfully. Fog Creek Software. Here’s the story:

 * More on Sabbaticals…
 * Command and Conquer and the Herd of Coconuts
 * Spring in Cambridge
 * What is the Work of Dogs in this Country?
 * Working on CityDesk, Part One
 * Working on CityDesk, Part Two
 * Working on CityDesk, Part Three
 * Working on CityDesk, Part Four
 * Working on CityDesk, Part Five
 * Rub a dub dub
 * New Server at Peer 1 Network
 * Finding an Office in New York City
 * Bionic Office
 * Colo Expansion Version 2.0
 * The Road to FogBugz 4.0: Part I
 * The Road to FogBugz 4.0: Part II
 * The Road to FogBugz 4.0: Part III
 * The Road to FogBugz 4.0: Part IV
 * The Road to FogBugz 4.0: Part V
 * Project Aardvark Midterm Report
 * How to Ship Anything
 * FogBugz 4½ and Subjective Well-Being
 * Copilot 2.0 ships!
 * A game of inches
 * Five whys
 * The new Fog Creek office
 * New, faster Copilot
 * Fruity treats, customization, and supersonics: FogBugz 7 is here
 * How Trello is different
 * Introducing HyperDev
 * Anil Dash is the new CEO of Fog Creek Software
 Proudly powered by WordPress
Since the 1990s, the web has been a publishing place for human-readable documents. Documents published on the web are in HTML. HTML has a little bit of… Read more “Progress on the Block Protocol” 
 
	You’ve probably seen web editors based on the idea of blocks. I’m typing this in WordPress, which has a little + button that brings up a long… Read more “Making the web better. With blocks!” 
 
	The other day I was talking to a young developer working on a code base with tons of COM code, and I told him that even before… Read more “Kinda a big announcement” 
 
	
	Sometimes simulating complex systems is the best way to understand them. Read more “HASH: a free, online platform for modeling the world”
Joel on Software
https://www.joelonsoftware.com/
https://davidwalsh.name/
David Walsh Blog
2023-05-09
David Walsh
<div><div><main><ul> <li><h2><a href="https://davidwalsh.name/restart-command-line">Restart Mac From Command Line</a></h2><p>Restarting and shutting down a computer remotely is a frequent task for remote system administrators. As someone that writes many shell scripts, I also find myself automating system restarts. Let's look at a few ways to restart Mac systems from command line!Restart a Local MacTo...</p></li><li><div>
 
 <p><a href="https://requestmetrics.com/">
 
 </a>
 </p>
 <p><a href="https://requestmetrics.com/">
 
 </a>
 </p>
</div>
</li><li><div><h2><a href="https://davidwalsh.name/bluesky-use-domain">How to Use Your Domain on Bluesky</a></h2><p><a href="https://blueskyweb.xyz/">Bluesky</a> is a hot new social networking platform that functions like Twitter from Twitter's original founder. New users are flooding into the platform as a respite from Elon Musk's vision of Twitter and the fumbles that have happened since his takeover. Upon signing up...</p></div></li><li><div><h2><a href="https://davidwalsh.name/10-great-multipurpose-wordpress-themes">10 Great Multipurpose WordPress Themes</a></h2><p>Multipurpose themes are flexible WordPress templates that can be used to create virtually any kind of website. They are often best-sellers, and because they are so popular there are a lot of them to choose from. Too many in fact unless you have time to spare...</p></div></li><li><div><h2><a href="https://davidwalsh.name/video-blur-faces">How to Blur Faces in a Video from Command Line</a></h2><p>Privacy is always incredibly important, especially with visual media where you may not have the permission of individuals in the video. If you're filming something in public, it's likely you'll catch someone's face who simply doesn't want or need to be identified. This recently...</p></div></li><li><div><h2><a href="https://davidwalsh.name/content-visibility">CSS content-visibility</a></h2><p>The CSS language is full of small gaps which are frustrating to navigate. Between CSS properties to hide a container and its contents, there is still room for improvement. <code>visibility: hidden</code> keeps height and width integrity while <code>display: none</code> on a container hides everything. You can...</p></div></li><li><div><h2><a href="https://davidwalsh.name/get-hardware-id">How to Get a Computer’s Hardware ID</a></h2><p>Cheating in online games is a huge issue these days -- just ask anyone playing PUBG. Cheaters aren't difficult for players to spot but vendors oftentimes don't do enough to punish these villains. Krafton recently announced they would start banning cheaters by hardware ID...</p></div></li><li><div><h2><a href="https://davidwalsh.name/array-group">JavaScript Array Group</a></h2><p>Managing, sorting, and manipulating data with JavaScript is a skill we've often delegated to third party libraries like lodash. As the JavaScript language progresses, however, these features eventually get. added to the JavaScript specification. Two such APIs for grouping of <code>Array</code> data are `<code>Array.prototype.group</code>...</p></div></li><li></li><li></li><li><div><h2><a href="https://davidwalsh.name/block-range-ip">How to Block a Range of IP Addresses</a></h2><p>As much as content creators want traffic to their website, there is such thing as the wrong type of traffic. Sometimes it's content scrapers, sometimes it's malicious bots; either way, it's important to know how to block problematic IPs from your site.To block a range...</p></div></li></ul><div>
 
 
 <p><a href="https://requestmetrics.com/">
 
 </a>
 </p>
 <p><a href="https://requestmetrics.com/">
 
 </a>
 </p>
 
</div><div><h2>Popular Features</h2><ul><li><div><h3><a href="https://davidwalsh.name/leaving-mozilla">39 Shirts – Leaving Mozilla</a></h3><p>In 2001 I had just graduated from a small town high school and headed off to a small town college. I found myself in the quaint computer lab where the substandard computers featured two browsers: Internet Explorer and Mozilla. It was this lab where I fell...</p></div></li><li></li><li><div><h3><a href="https://davidwalsh.name/new-office">Welcome to My New Office</a></h3><p>My first professional web development was at a small print shop where I sat in a windowless cubical all day. I suffered that boxed in environment for almost five years before I was able to find a remote job where I worked from home. The first...</p></div></li><li><div><h3><a href="https://davidwalsh.name/retropie-graphical-guide">How to Create a RetroPie on Raspberry Pi – Graphical Guide</a></h3><p>Today we get to play amazing games on our super powered game consoles, PCs, VR headsets, and even mobile devices.  While I enjoy playing new games these days, I do long for the retro gaming systems I had when I was a kid: the original Nintendo...</p></div></li><li><div><h3><a href="https://davidwalsh.name/conquering-impostor-syndrome">Conquering Impostor Syndrome</a></h3><p>Two years ago I documented my struggles with <a href="https://davidwalsh.name/impostor-syndrome">Imposter Syndrome</a> and the response was immense.  I received messages of support and commiseration from new web developers, veteran engineers, and even persons of all experience levels in other professions.  I've even caught myself reading the post...</p></div></li><li><div><h3><a href="https://davidwalsh.name/being-a-dev-dad">Being a Dev Dad</a></h3><p>I get asked loads of questions every day but I'm always surprised that they're rarely questions about code or even tech -- many of the questions I get are more about non-dev stuff like <a href="https://davidwalsh.name/office">what my office is like</a>, what software I use, and oftentimes...</p></div></li><li><div><h3><a href="https://davidwalsh.name/promises">JavaScript Promise API</a></h3><p>While synchronous code is easier to follow and debug, async is generally better for performance and flexibility. Why "hold up the show" when you can trigger numerous requests at once and then handle them when each is ready?  Promises are becoming a big part of the JavaScript world...</p></div></li></ul></div>
</main></div></div>||||I|||| * Home
 * Main Content
 DWB
 * Home
 * Tutorials
 * Features
 * Demos
 * Topics
 * The Blog
 * Recent Tutorials
 + Restart Mac From Command Line
 + How to Use Your Domain on Bluesky
 + 10 Great Multipurpose WordPress Themes
 + How to Blur Faces in a Video from Command Line
 + CSS content-visibility
 * Recent Features
 + 39 Shirts - Leaving Mozilla
 + Interview with a Pornhub Web Developer
 + Welcome to My New Office
 + How to Create a RetroPie on Raspberry Pi - Graphical Guide
 + Conquering Impostor Syndrome
 * Interactive Demos
 + Chris Coyier’s Favorite CodePen Demos IV
 + prefers-color-scheme: CSS Media Query
 + CSS Ellipsis Beginning of String
 + Vertically Centering with Flexbox
 + Creating Spacers with Flexbox
 * Popular Topics
 + HTML5
 + CSS Animations
 + Firefox OS
 + jQuery
 + MooTools
 + PHP
 + CSS3
 + WordPress
 + Mobile
 + SEO
 + JavaScript
 + Dojo Toolkit
 * David Walsh Blog
 + About David Walsh
 + Contact and Advertise
 + Developer Deals
 + Mozilla
 * 
 * 
 * 
 * 
 * 
 * 
 Search
 Popular:
 * JavaScript Promises
 * fetch API
 * React.js
 * Cache API
 * ES6 Features
 * Node.js
 * JavaScript
 * jQuery
 David Walsh Blog - Home

 David Walsh Blog

 * By David Walsh May 9, 2023

 Restart Mac From Command Line

 Restarting and shutting down a computer remotely is a frequent task for remote system administrators. As someone that writes many shell scripts, I also find myself automating system restarts. Let's look at a few ways to restart Mac systems from command line!Restart a Local MacTo...

 * 
 * By David Walsh May 8, 2023

 How to Use Your Domain on Bluesky

 Bluesky is a hot new social networking platform that functions like Twitter from Twitter's original founder. New users are flooding into the platform as a respite from Elon Musk's vision of Twitter and the fumbles that have happened since his takeover. Upon signing up...

 * By David Walsh May 2, 2023

 10 Great Multipurpose WordPress Themes

 Multipurpose themes are flexible WordPress templates that can be used to create virtually any kind of website. They are often best-sellers, and because they are so popular there are a lot of them to choose from. Too many in fact unless you have time to spare...

 * By David Walsh May 1, 2023

 How to Blur Faces in a Video from Command Line

 Privacy is always incredibly important, especially with visual media where you may not have the permission of individuals in the video. If you're filming something in public, it's likely you'll catch someone's face who simply doesn't want or need to be identified. This recently...

 * By David Walsh April 27, 2023

 CSS content-visibility

 The CSS language is full of small gaps which are frustrating to navigate. Between CSS properties to hide a container and its contents, there is still room for improvement. visibility: hidden keeps height and width integrity while display: none on a container hides everything. You can...

 * By David Walsh April 18, 2023

 How to Get a Computer’s Hardware ID

 Cheating in online games is a huge issue these days -- just ask anyone playing PUBG. Cheaters aren't difficult for players to spot but vendors oftentimes don't do enough to punish these villains. Krafton recently announced they would start banning cheaters by hardware ID...

 * By David Walsh April 17, 2023

 JavaScript Array Group

 Managing, sorting, and manipulating data with JavaScript is a skill we've often delegated to third party libraries like lodash. As the JavaScript language progresses, however, these features eventually get. added to the JavaScript specification. Two such APIs for grouping of Array data are `Array.prototype.group...

 * By David Walsh April 13, 2023

 How to Get a Base64 Version of a File From Command Line

 A while back I wrote an article on how to Convert Image to Data URI with JavaScript. It's a neat trick developers can use for any number of reasons. Instead of abusing canvas, however, why not simply get the base64 data from...

 * By David Walsh April 10, 2023

 How to Get Mac Battery Level from Command Line

 I'm a big fan of having as much information as I can get within the command line. I couldn't go without knowing which git branch I'm on, for example. Another important piece of information I like having is my current battery percentage.To get...

 * By David Walsh March 27, 2023

 How to Block a Range of IP Addresses

 As much as content creators want traffic to their website, there is such thing as the wrong type of traffic. Sometimes it's content scrapers, sometimes it's malicious bots; either way, it's important to know how to block problematic IPs from your site.To block a range...

 Recent TutorialsFeatured TutorialsIncredible Demos

 Popular Features

 * By David Walsh October 6, 2020

 39 Shirts – Leaving Mozilla

 In 2001 I had just graduated from a small town high school and headed off to a small town college. I found myself in the quaint computer lab where the substandard computers featured two browsers: Internet Explorer and Mozilla. It was this lab where I fell...

 * By David Walsh October 7, 2019

 Interview with a Pornhub Web Developer

 Regardless of your stance on pornography, it would be impossible to deny the massive impact the adult website industry has had on pushing the web forward. From pushing the browser's video limits to pushing ads through WebSocket so ad blockers don't detect them, you have...

 * By David Walsh February 19, 2019

 Welcome to My New Office

 My first professional web development was at a small print shop where I sat in a windowless cubical all day. I suffered that boxed in environment for almost five years before I was able to find a remote job where I worked from home. The first...

 * By David Walsh September 21, 2017

 How to Create a RetroPie on Raspberry Pi – Graphical Guide

 Today we get to play amazing games on our super powered game consoles, PCs, VR headsets, and even mobile devices. While I enjoy playing new games these days, I do long for the retro gaming systems I had when I was a kid: the original Nintendo...

 * By David Walsh September 18, 2017

 Conquering Impostor Syndrome

 Two years ago I documented my struggles with Imposter Syndrome and the response was immense. I received messages of support and commiseration from new web developers, veteran engineers, and even persons of all experience levels in other professions. I've even caught myself reading the post...

 * By David Walsh June 29, 2016

 Being a Dev Dad

 I get asked loads of questions every day but I'm always surprised that they're rarely questions about code or even tech -- many of the questions I get are more about non-dev stuff like what my office is like, what software I use, and oftentimes...

 * By David Walsh November 2, 2015

 JavaScript Promise API

 While synchronous code is easier to follow and debug, async is generally better for performance and flexibility. Why "hold up the show" when you can trigger numerous requests at once and then handle them when each is ready? Promises are becoming a big part of the JavaScript world...

 All FeaturesIncredible Demos
 * 
 * 
 * 
 * 
 * 
 * 

 © David Walsh 2007-2023. All code MIT license . Hosting by Media Temple .

Restart Mac From Command Line  Restarting and shutting down a computer remotely is a frequent task for remote system administrators. As someone that writes many shell scripts, I also find myself automating system restarts. Let's look at a few ways to restart Mac systems from command line!Restart a Local MacTo...   
 
  
 
 
 
  
 
 
 
    How to Use Your Domain on Bluesky   Bluesky is a hot new social networking platform that functions like Twitter from Twitter's original founder. New users are flooding into the platform as a respite from Elon Musk's vision of Twitter and the fumbles that have happened since his takeover. Upon signing up...      10 Great Multipurpose WordPress Themes  Multipurpose themes are flexible WordPress templates that can be used to create virtually any kind of website. They are often best-sellers, and because they are so popular there are a lot of them to choose from. Too many in fact unless you have time to spare...      How to Blur Faces in a Video from Command Line  Privacy is always incredibly important, especially with visual media where you may not have the permission of individuals in the video. If you're filming something in public, it's likely you'll catch someone's face who simply doesn't want or need to be identified. This recently...      CSS content-visibility  The CSS language is full of small gaps which are frustrating to navigate. Between CSS properties to hide a container and its contents, there is still room for improvement. visibility: hidden keeps height and width integrity while display: none on a container hides everything. You can...      How to Get a Computer’s Hardware ID  Cheating in online games is a huge issue these days -- just ask anyone playing PUBG. Cheaters aren't difficult for players to spot but vendors oftentimes don't do enough to punish these villains. Krafton recently announced they would start banning cheaters by hardware ID...      JavaScript Array Group  Managing, sorting, and manipulating data with JavaScript is a skill we've often delegated to third party libraries like lodash. As the JavaScript language progresses, however, these features eventually get. added to the JavaScript specification. Two such APIs for grouping of Array data are `Array.prototype.group...          How to Block a Range of IP Addresses  As much as content creators want traffic to their website, there is such thing as the wrong type of traffic. Sometimes it's content scrapers, sometimes it's malicious bots; either way, it's important to know how to block problematic IPs from your site.To block a range...    
 
 
  
 
 
 
  
 
 
 
 
  Popular Features     39 Shirts – Leaving Mozilla  In 2001 I had just graduated from a small town high school and headed off to a small town college. I found myself in the quaint computer lab where the substandard computers featured two browsers: Internet Explorer and Mozilla. It was this lab where I fell...        Welcome to My New Office  My first professional web development was at a small print shop where I sat in a windowless cubical all day. I suffered that boxed in environment for almost five years before I was able to find a remote job where I worked from home. The first...      How to Create a RetroPie on Raspberry Pi – Graphical Guide  Today we get to play amazing games on our super powered game consoles, PCs, VR headsets, and even mobile devices.  While I enjoy playing new games these days, I do long for the retro gaming systems I had when I was a kid: the original Nintendo...      Conquering Impostor Syndrome  Two years ago I documented my struggles with Imposter Syndrome and the response was immense.  I received messages of support and commiseration from new web developers, veteran engineers, and even persons of all experience levels in other professions.  I've even caught myself reading the post...      Being a Dev Dad  I get asked loads of questions every day but I'm always surprised that they're rarely questions about code or even tech -- many of the questions I get are more about non-dev stuff like what my office is like, what software I use, and oftentimes...      JavaScript Promise API  While synchronous code is easier to follow and debug, async is generally better for performance and flexibility. Why "hold up the show" when you can trigger numerous requests at once and then handle them when each is ready?  Promises are becoming a big part of the JavaScript world...
David Walsh Blog
https://davidwalsh.name/
https://tenderlovemaking.com/
Vim, tmux, and Fish
2023-01-18
None
<div><div>
 <p>Encoding issues don’t seem to happen frequently, but that is a blessing and a curse.
It’s great not to fix them very frequently, but when you do need to fix them, lack
of experience can leave you feeling lost.</p>
<p>This post is meant to be a sort of guide about what to do when you encounter different types
of encoding errors in Ruby.
First we’ll cover what an encoding object is, then we’ll look at common encoding exceptions
and how to fix them.</p>
<h2>What are String encodings in Ruby?</h2>
<p>In Ruby, strings are a combination of an array of bytes, and an encoding object.
We can access the encoding object on the string by calling <code>encoding</code> on the
string object.</p>
<p>For example:</p>
<pre><code>&gt;&gt; x = 'Hello World'
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
</code></pre>
<p>In my environment, the default encoding object associated with a string us the “UTF-8” encoding object.
A graph of the object relationship looks something like this:</p>
<p></p>
<h2>Changing a String’s Encoding</h2>
<p>We can change encoding by two different methods:</p>
<ul>
 <li>String#force_encoding</li>
 <li>String#encode</li>
</ul>
<p>The <code>force_encoding</code> method will mutate the string object and only change which encoding object the string points to.
It does nothing to the bytes of the string, it merely changes the encoding object associated with the string.
Here we can see that the return value of <code>encoding</code> changes after we call the <code>force_encode</code> method:</p>
<pre><code>&gt;&gt; x = 'Hello World'
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.force_encoding "US-ASCII"
=&gt; "Hello World"
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:US-ASCII&gt;
</code></pre>
<p>The <code>encode</code> method will create a new string based on the bytes of the old string and associate the encoding object with the new string.</p>
<p>Here we can see that the encoding of <code>x</code> remains the same, and
calling <code>encode</code> returns a new string <code>y</code> which is associated with the new encoding:</p>
<pre><code>&gt;&gt; x = 'Hello World'
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; y = x.encode("US-ASCII")
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; y.encoding
=&gt; #&lt;Encoding:US-ASCII&gt;
</code></pre>
<p>Here is a visualization of the difference:</p>
<p></p>
<p>Calling <code>force_encoding</code> mutates the original string, where <code>encode</code> creates a new string with a different encoding.
Translating a string from one encoding to another is probably the “normal” use of encodings.
However, developers will rarely call the <code>encode</code> method because Ruby will typically handle any necessary translations automatically.
It’s probably more common to call the <code>force_encoding</code> method, and that is because strings can be associated with the <em>wrong</em> encoding.</p>
<h2>Strings Can Have the Wrong Encoding</h2>
<p>Strings can be associated with the wrong encoding object, and that is the source of most if not all encoding related exceptions.
Let’s look at an example:</p>
<pre><code>&gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.valid_encoding?
=&gt; false
</code></pre>
<p>In this case, Ruby associated the string <code>"Hello \x93\xfa\x96\x7b"</code> with the default encoding UTF-8.
However, many of the bytes in the string are not valid Unicode characters.
We can check if the string is associated with a valid encoding object by calling <code>valid_encoding?</code> method.
The <code>valid_encoding?</code> method will scan all bytes to see if they are valid for that particular encoding object.</p>
<p>So how do we fix this?
The answer depends on the situation.
We need to think about where the data came from and where the data is going.
Let’s say we’ll display this string on a webpage, but we do not know the correct encoding for the string.
In that case we probably want to make sure the string is valid UTF-8, but since we don’t know the correct encoding for the string, our only choice is to remove the bad bytes from the string.</p>
<p>We can remove the unknown bytes by using the <code>scrub</code> method:</p>
<pre><code>&gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.valid_encoding?
=&gt; false
&gt;&gt; y = x.scrub
&gt;&gt; y
=&gt; "Hello ï¿½ï¿½ï¿½{"
&gt;&gt; y.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; y.valid_encoding?
=&gt; true
</code></pre>
<p>The <code>scrub</code> method will return a new string associated with the encoding but with all of the invalid bytes replaced by a replacement character, the diamond question mark thing.</p>
<p>What if we do know the encoding of the source string?
Actually the example above is using a string that’s encoding using Shift JIS.
Let’s say we know the encoding, and we want to display the string on a webpage.
In that case we tag the string by using <code>force_encoding</code>, and transcode to UTF-8:</p>
<pre><code>&gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.force_encoding "Shift_JIS"
=&gt; "Hello \x{93FA}\x{967B}"
&gt;&gt; x.valid_encoding?
=&gt; true
&gt;&gt; x.encode "UTF-8" # display as UTF-8
=&gt; "Hello æ¥æ¬"
</code></pre>
<p>The most important thing to think about when dealing with encoding issues is “where did this data come from?” and “what will we do with this data?”
Answering those two questions will drive all decisions about which encoding to use with which string.</p>
<h2>Encoding Depends on the Context</h2>
<p>Before we look at some common errors and their remediation, let’s look at one more example of the encoding context dependency.
In this example, we’ll use some user input as a cache key, but we’ll also display the user input on a webpage.
We’re going to use our source data (the user input) in two places: as a cache key, and something to display on a web page.</p>
<p>Here’s the code:</p>
<div><pre>require "digest/md5"
require "cgi"
def make_checksum string
 Digest::MD5.hexdigest string
end
def display_on_web string
 string.gsub(/&gt;/, "&amp;gt;")
end
x = "Hello \x93\xfa\x96\x7b"
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?
p display_on_web x
p make_checksum x
</pre></div>
<p>If we run this code, we’ll get an exception:</p>
<pre><code>$ ruby thing.rb
{:ENCODING=&gt;#&lt;Encoding:UTF-8&gt;}
{:VALID_ENCODING=&gt;false}
Traceback (most recent call last):
 2: from thing.rb:20:in `&lt;main&gt;'
 1: from thing.rb:12:in `display_on_web'
thing.rb:12:in `gsub': invalid byte sequence in UTF-8 (ArgumentError)
</code></pre>
<p>The problem is that we have a string of unknown input with bytes that are not valid UTF-8 characters.
We know we want to display this string on a UTF-8 encoded webpage, so lets scrub the string:</p>
<div><pre>require "digest/md5"
require "cgi"
def make_checksum string
 Digest::MD5.hexdigest string
end
def display_on_web string
 string.gsub(/&gt;/, "&amp;gt;")
end
x = "Hello \x93\xfa\x96\x7b".scrub
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?
p display_on_web x
p make_checksum x
</pre></div>
<p>Now when we run the program, the output is like this:</p>
<pre><code>$ ruby thing.rb
{:ENCODING=&gt;#&lt;Encoding:UTF-8&gt;}
{:VALID_ENCODING=&gt;true}
"Hello ï¿½ï¿½ï¿½{"
"4dab6f63b4d3ae3279345c9df31091eb"
</code></pre>
<p>Great! We’ve build some HTML and generated a checksum.
Unfortunately there is a bug in this code (of course the mere fact that we’ve written code means there’s a bug! lol)
Let’s introduce a second user input string with slightly different bytes than the first input string:</p>
<div><pre>require "digest/md5"
require "cgi"
def make_checksum string
 Digest::MD5.hexdigest string
end
def display_on_web string
 string.gsub(/&gt;/, "&amp;gt;")
end
x = "Hello \x93\xfa\x96\x7b".scrub
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?
p display_on_web x
p make_checksum x
y = "Hello \x94\xfa\x97\x7b".scrub
p ENCODING: y.encoding
p VALID_ENCODING: y.valid_encoding?
p display_on_web y
p make_checksum y
</pre></div>
<p>Here is the output from the program:</p>
<pre><code>$ ruby thing.rb
{:ENCODING=&gt;#&lt;Encoding:UTF-8&gt;}
{:VALID_ENCODING=&gt;true}
"Hello ï¿½ï¿½ï¿½{"
"4dab6f63b4d3ae3279345c9df31091eb"
{:ENCODING=&gt;#&lt;Encoding:UTF-8&gt;}
{:VALID_ENCODING=&gt;true}
"Hello ï¿½ï¿½ï¿½{"
"4dab6f63b4d3ae3279345c9df31091eb"
</code></pre>
<p>The program works in the sense that there is no exception.
But both user input strings have the same checksum despite the fact that the original strings clearly have different bytes!
So what is the correct fix for this program?
Again, we need to think about the source of the data (where did it come from), as well as what we will do with it (where it is going).
In this case we have one source, from a user, and the user provided us with no encoding information.
In other words, the encoding information of the source data is unknown, so we can only treat it as a sequence of bytes.
We have two output cases, one is a UTF-8 HTML the other output is <em>the input</em> to our checksum function.
The HTML requires that our string be UTF-8 so making the string valid UTF-8, in other words “scrubbing” it, before displaying makes sense.
However, our checksum function requires seeing the original bytes of the string.
Since the checksum is only concerned with the bytes in the string, any encoding including an invalid encoding will work.
It’s nice to make sure all our strings have valid encodings though, so we’ll fix this example such that everything has a valid encoding.</p>
<div><pre>require "digest/md5"
require "cgi"
def make_checksum string
 Digest::MD5.hexdigest string
end
def display_on_web string
 string.gsub(/&gt;/, "&amp;gt;")
end
x = "Hello \x93\xfa\x96\x7b".b
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?
p display_on_web x.encode("UTF-8", undef: :replace)
p make_checksum x
y = "Hello \x94\xfa\x97\x7b".b
p ENCODING: y.encoding
p VALID_ENCODING: y.valid_encoding?
p display_on_web y.encode("UTF-8", undef: :replace)
p make_checksum y
</pre></div>
<p>Here is the output of the program:</p>
<pre><code>$ ruby thing.rb
{:ENCODING=&gt;#&lt;Encoding:ASCII-8BIT&gt;}
{:VALID_ENCODING=&gt;true}
"Hello ï¿½ï¿½ï¿½{"
"96cf6db2750fd4d2488fac57d8e4d45a"
{:ENCODING=&gt;#&lt;Encoding:ASCII-8BIT&gt;}
{:VALID_ENCODING=&gt;true}
"Hello ï¿½ï¿½ï¿½{"
"b92854c0db4f2c2c20eff349a9a8e3a0"
</code></pre>
<p>To fix our program, we’ve changed a couple things.
First we tagged the string of unknown encoding as “binary” by using the <code>.b</code> method.
The <code>.b</code> method returns a new string that is associated with the <code>ASCII-8BIT</code> encoding.
The name <code>ASCII-8BIT</code> is somewhat confusing because it has the word “ASCII” in it.
It’s better to think of this encoding as either “unknown” or “binary data”.
Unknown meaning we have some data that may have a valid encoding, but we don’t know what it is.
Or binary data, as in the bytes read from a JPEG file or some such binary format.
Anyway, we pass the binary string in to the checksum function because the checksum only cares about the bytes in the string, not about the encoding.</p>
<p>The second change we made is to call <code>encode</code> with the encoding we want (UTF-8) along with <code>undef: :replace</code> meaning that any time Ruby encounters bytes it doesn’t know how to convert to the target encoding, it will replace them with the replacement character (the diamond question thing).</p>
<p>SIDE NOTE: This is probably not important, but it is fun!
We can specify what Ruby uses for replacing unknown bytes.
Here’s an example:</p>
<pre><code>&gt;&gt; x = "Hello \x94\xfa\x97\x7b".b
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:ASCII-8BIT&gt;
&gt;&gt; x.encode("UTF-8", undef: :replace, replace: "Aaron")
=&gt; "Hello AaronAaronAaron{"
&gt;&gt; x.encode("UTF-8", undef: :replace, replace: "ð¤£")
=&gt; "Hello ð¤£ð¤£ð¤£{"
&gt;&gt; [_.encoding, _.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
</code></pre>
<p>Now lets take a look at some common encoding errors in Ruby and what to do about them.</p>
<h2><code>Encoding::InvalidByteSequenceError</code></h2>
<p>This exception occurs when Ruby needs to examine the bytes in a string and the bytes do not match the encoding.
Here is an example of this exception:</p>
<pre><code>&gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.encode "UTF-16"
Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 2: from (irb):4
 1: from (irb):4:in `encode'
Encoding::InvalidByteSequenceError ("\x93" on UTF-8)
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.valid_encoding?
=&gt; false
</code></pre>
<p>The string <code>x</code> contains bytes that aren’t valid UTF-8, yet it is associated with the UTF-8 encoding object.
When we try to convert <code>x</code> to UTF-16, an exception occurs.</p>
<h3>How to fix <code>Encoding::InvalidByteSequenceError</code></h3>
<p>Like most encoding issues, our string <code>x</code> is tagged with the wrong encoding.
The way to fix this issue is to tag the string with the correct encoding.
But what is the correct encoding?
To figure out the correct encoding, you need to know where the string came from.
For example if the string came from a Mime attachment, the Mime attachment should specify the encoding (or the RFC will tell you).</p>
<p>In this case, the string is a valid Shift JIS string, but I know that because I looked up the bytes and manually entered them.
So we’ll tag this as Shift JIS, and the exception goes away:</p>
<pre><code>&gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.force_encoding "Shift_JIS"
=&gt; "Hello \x{93FA}\x{967B}"
&gt;&gt; x.encode "UTF-16"
=&gt; "\uFEFFHello \u65E5\u672C"
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:Shift_JIS&gt;
&gt;&gt; x.valid_encoding?
=&gt; true
</code></pre>
<p>If you don’t know the source of the string, an alternative solution is to tag as UTF-8 and then scrub the bytes:</p>
<pre><code>&gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.force_encoding "UTF-8"
=&gt; "Hello \x93\xFA\x96{"
&gt;&gt; x.scrub!
=&gt; "Hello ï¿½ï¿½ï¿½{"
&gt;&gt; x.encode "UTF-16"
=&gt; "\uFEFFHello \uFFFD\uFFFD\uFFFD{"
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.valid_encoding?
=&gt; true
</code></pre>
<p>Of course this works, but it means that you’ve lost data.
The best solution is to figure out what the encoding of the string <em>should</em> be depending on its source and tag it with the correct encoding.</p>
<h2><code>Encoding::UndefinedConversionError</code></h2>
<p>This exception occurs when a string of one encoding can’t be converted to another encoding.</p>
<p>Here is an example:</p>
<pre><code>&gt;&gt; x = "å\u2160"
&gt;&gt; x
=&gt; "åâ "
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.valid_encoding?
=&gt; true
&gt;&gt; x.encode "Shift_JIS"
Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 2: from (irb):23
 1: from (irb):23:in `encode'
Encoding::UndefinedConversionError (U+2160 from UTF-8 to Shift_JIS)
</code></pre>
<p>In this example, we have two characters: “å”, and the Roman numeral 1 (“â ”).
Unicode Roman numeral 1 cannot be converted to Shift JIS because there are <em>two</em> codepoints that represent that character in Shift JIS.
This means the conversion is ambiguous, so Ruby will raise an exception.</p>
<h3>How to fix <code>Encoding::UndefinedConversionError</code></h3>
<p>Our original string is correctly tagged as UTF-8, but we need to convert to Shift JIS.
In this case we’ll use a replacement character when converting to Shift JIS:</p>
<pre><code>&gt;&gt; x = "å\u2160"
&gt;&gt; y = x.encode("Shift_JIS", undef: :replace)
&gt;&gt; y
=&gt; "\x{8E6C}?"
&gt;&gt; y.encoding
=&gt; #&lt;Encoding:Shift_JIS&gt;
&gt;&gt; y.valid_encoding?
=&gt; true
&gt;&gt; y.encode "UTF-8"
=&gt; "å?"
</code></pre>
<p>We were able to convert to Shift JIS, but we did lose some data.</p>
<h2><code>ArgumentError</code></h2>
<p>When a string contains invalid bytes, sometimes Ruby will raise an <code>ArgumentError</code> exception:</p>
<pre><code>&gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.downcase
Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 2: from (irb):34
 1: from (irb):34:in `downcase'
ArgumentError (input string invalid)
&gt;&gt; x.gsub(/ello/, "i")
Traceback (most recent call last):
 6: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 3: from (irb):34
 2: from (irb):35:in `rescue in irb_binding'
 1: from (irb):35:in `gsub'
ArgumentError (invalid byte sequence in UTF-8)
</code></pre>
<p>Again we use our incorrectly tagged Shift JIS string.
Calling <code>downcase</code> or <code>gsub</code> both result in an <code>ArgumentError</code>.
I personally think these exceptions are not great.
We didn’t pass anything to <code>downcase</code>, so why is it an <code>ArgumentError</code>?
There is nothing wrong with the arguments we passed to <code>gsub</code>, so why is it an <code>ArgumentError</code>?
Why does one say “input string invalid” where the other gives us a slightly more helpful exception of “invalid byte sequence in UTF-8”?
I think these should both result in <code>Encoding::InvalidByteSequenceError</code> exceptions, as it’s a problem with the encoding, not the arguments.</p>
<p>Regardless, these errors both stem from the fact that the Shift JIS string is incorrectly tagged as UTF-8.</p>
<h3>Fixing <code>ArgumentError</code></h3>
<p>Fixing this issue is just like fixing <code>Encoding::InvalidByteSequenceError</code>.
We need to figure out the correct encoding of the source string, then tag the source string with that encoding.
If the encoding of the source string is truly unknown, scrub it.</p>
<pre><code>&gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.force_encoding "Shift_JIS"
=&gt; "Hello \x{93FA}\x{967B}"
&gt;&gt; x.downcase
=&gt; "hello \x{93FA}\x{967B}"
&gt;&gt; x.gsub(/ello/, "i")
=&gt; "Hi \x{93FA}\x{967B}"
</code></pre>
<h2><code>Encoding::CompatibilityError</code></h2>
<p>This exception occurs when we try to combine strings of two different encodings and those encodings are incompatible.
For example:</p>
<pre><code>&gt;&gt; x = "å\u2160"
&gt;&gt; y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
&gt;&gt; [x.encoding, x.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
&gt;&gt; [y.encoding, y.valid_encoding?]
=&gt; [#&lt;Encoding:Shift_JIS&gt;, true]
&gt;&gt; x + y
Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 2: from (irb):50
 1: from (irb):50:in `+'
Encoding::CompatibilityError (incompatible character encodings: UTF-8 and Shift_JIS)
</code></pre>
<p>In this example we have a valid UTF-8 string and a valid Shift JIS string.
However, these two encodings are not compatible, so we get an exception when combining.</p>
<h3>Fixing <code>Encoding::CompatibilityError</code></h3>
<p>To fix this exception, we need to manually convert one string to a new string that has a compatible encoding.
In the case above, we can choose whether we want the output string to be UTF-8 or Shift JIS, and then call <code>encode</code> on the appropriate string.</p>
<p>In the case we want UTF-8 output, we can do this:</p>
<pre><code>&gt;&gt; x = "å"
&gt;&gt; y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
&gt;&gt; x + y.encode("UTF-8")
=&gt; "åHello æ¥æ¬"
&gt;&gt; [_.encoding, _.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
</code></pre>
<p>If we wanted Shift JIS, we could do this:</p>
<pre><code>&gt;&gt; x = "å"
&gt;&gt; y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
&gt;&gt; x.encode("Shift_JIS") + y
=&gt; "\x{8E6C}Hello \x{93FA}\x{967B}"
&gt;&gt; [_.encoding, _.valid_encoding?]
=&gt; [#&lt;Encoding:Shift_JIS&gt;, true]
</code></pre>
<p>Another possible solution is to scrub bytes and concatenate, but again that results in data loss.</p>
<h3>What is a compatible encoding?</h3>
<p>If there are incompatible encodings, there must be compatible encodings too (at least I would think that).
Here is an example of compatible encodings:</p>
<pre><code>&gt;&gt; x = "Hello World!".force_encoding "US-ASCII"
&gt;&gt; [x.encoding, x.valid_encoding?]
=&gt; [#&lt;Encoding:US-ASCII&gt;, true]
&gt;&gt; y = "ããã«ã¡ã¯"
&gt;&gt; [y.encoding, y.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
&gt;&gt; y + x
=&gt; "ããã«ã¡ã¯Hello World!"
&gt;&gt; [_.encoding, _.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
&gt;&gt; x + y
=&gt; "Hello World!ããã«ã¡ã¯"
&gt;&gt; [_.encoding, _.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
</code></pre>
<p>The <code>x</code> string is encoded with “US ASCII” encoding and the <code>y</code> string UTF-8.
US ASCII is fully compatible with UTF-8, so even though these two strings have different encoding, concatenation works fine.</p>
<p>String literals may default to UTF-8, but some functions will return US ASCII encoded strings.
For example:</p>
<pre><code>&gt;&gt; require "digest/md5"
=&gt; true
&gt;&gt; Digest::MD5.hexdigest("foo").encoding
=&gt; #&lt;Encoding:US-ASCII&gt;
</code></pre>
<p>A hexdigest will only ever contain ASCII characters, so the implementation tags the returned string as US-ASCII.</p>
<h2>Encoding Gotchas</h2>
<p>Let’s look at a couple encoding gotcha’s.</p>
<h3>Infectious Invalid Encodings</h3>
<p>When a string is incorrectly tagged, Ruby will typically only raise an exception when it needs to actually examine the bytes.
Here is an example:</p>
<pre><code>&gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.valid_encoding?
=&gt; false
&gt;&gt; x + "ã»ã"
=&gt; "Hello \x93\xFA\x96{ã»ã"
&gt;&gt; y = _
&gt;&gt; y
=&gt; "Hello \x93\xFA\x96{ã»ã"
&gt;&gt; [y.encoding, y.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, false]
</code></pre>
<p>Again we have the incorrectly tagged Shift JIS string.
We’re able to append a correctly tagged UTF-8 string and no exception is raised.
Why is that?
Ruby assumes that if both strings have the same encoding, there is no reason to validate the bytes in either string so it will just append them.
That means we can have an incorrectly tagged string “infect” what would otherwise be correctly tagged UTF-8 strings.
Say we have some code like this:</p>
<div><pre>def append string
 string + "ã»ã"
end
p append("ã»ã").valid_encoding? 
p append("Hello \x93\xfa\x96\x7b").valid_encoding? 
</pre></div>
<p>When debugging this code, we may be tempted to think the problem is in the <code>append</code> method.
But actually the issue is with <em>the caller</em>.
The caller is passing in incorrectly tagged strings, and unfortunately we might not get an exception until the return value of <code>append</code> is used somewhere far away.</p>
<h3>ASCII-8BIT is Special</h3>
<p>Sometimes ASCII-8BIT is considered to be a “compatible” encoding and sometimes it isn’t.
Here is an example:</p>
<pre><code>&gt;&gt; x = "\x93\xfa\x96\x7b".b
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:ASCII-8BIT&gt;
&gt;&gt; y = "ã»ã"
&gt;&gt; y + x
Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 2: from (irb):89
 1: from (irb):89:in `+'
Encoding::CompatibilityError (incompatible character encodings: UTF-8 and ASCII-8BIT)
</code></pre>
<p>Here we have a binary string stored in <code>x</code>.
Maybe it came from a JPEG file or something (it didn’t, I just typed it in!)
When we try to concatenate the binary string with the UTF-8 string, we get an exception.
But this may actually be an exception we want!
It doesn’t make sense to be concatenating some JPEG data with an actual string we want to view, so it’s <em>good</em> we got an exception here.</p>
<p>Now here is the same code, but with the contents of <code>x</code> changed somewhat:</p>
<pre><code>&gt;&gt; x = "Hello World".b
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:ASCII-8BIT&gt;
&gt;&gt; y = "ã»ã"
&gt;&gt; y + x
=&gt; "ã»ãHello World"
</code></pre>
<p>We have the same code with the same encodings at play.
The only thing that changed is the actual contents of the <code>x</code> string.</p>
<p>When Ruby concatenates ASCII-8BIT strings, it will examine the contents of that string.
If all bytes in the string are ASCII characters, it will treat it as a US-ASCII string and consider it to be “compatible”.
If the string contains non-ASCII characters, it will consider it to be incompatible.</p>
<p>This means that if you had read some data from your JPEG, and that data happened to all be ASCII characters, you would not get an exception even though maybe you really wanted one.</p>
<p>In my personal opinion, concatenating an ASCII-8BIT string with anything besides another ASCII-8BIT string should be an exception.</p>
<p>Anyway, this is all I feel like writing today. I hope you have a good day, and remember to check your encodings!</p>
 </div>||||I|||| Vim, tmux, and Fish

 2023-01-18 @ 11:23

 I do most of my text editing with MacVim, but when I pair with people I like to use tmate. tmate is just an easy way to connect tmux sessions with a remote person. But this means that I go from coding in a GUI to coding in a terminal. Normally this wouldn’t be a problem, but I had made a Fish alias that would open the MacVim GUI every time I typed vim in the terminal. Of course when I’m pairing via tmate, the other people cannot see the GUI, so I would have to remember a different command to open Vim.

 Today I did about 10min of research to fix this problem and came up with the following Fish command:

 $ cat .config/fish/functions/vim.fish 
 function vim --wraps='vim' --description 'open Vim'
 if set -q TMUX # if we're in a TMUX session, open in terminal
 command vim $argv
 else
 # Otherwise open macvim
 open -a MacVim $argv; 
 end
 end
 

 All it does is open terminal Vim if I’m in a TMUX session, otherwise it opens the MacVim GUI.

 Instead of putting up with this frustration for such a long time, I should have taken the 10 min required to fix the situation. This was a good reminder for me, and hopefully I’ll be better about it in the future!

 read more »

 In Memory of a Giant

 2022-12-07 @ 07:08

 The Ruby community has lost a giant. As a programmer, I always feel as if I’m standing on the shoulders of giants. Chris Seaton was one of those giants.

 I’ve been working at the same company as Chris for the past 2 years. However, I first met him through the open source world many years ago. He was working on a Ruby implementation called TruffleRuby, and got his PhD in Ruby. Can you believe that? A PhD in Ruby? I’d never heard of such a thing. My impression was that nobody in academia cared about Ruby, but here was Chris, the Ruby Doctor. I was impressed.

 Patience

 As a college dropout, I’ve always felt underqualified. Embarrassment about my lack of knowledge and credentials has driven me to study hard on my own time. But Chris never once made me feel out of place. Any time I had questions, without judgement, he would take the time to explain things to me.

 I’ve always looked up to Chris. I was at a bar in London with a few coworkers. We started talking about age, and I found out that Chris was much younger than me. I said “You’re so smart and accomplished! How can I possibly catch up to you?” Chris said “Don’t worry, I’ll just tell you everything I know!”

 Puns

 My team is fully remote, so every Friday we have a team meeting over video to just hang out and talk about stuff. Eventually I’ll make a really great pun, most people will sigh, Kevin Menard will get angry, and Chris would just be straight faced. No reaction from Chris. Every. Single. Time.

 One time, someone asked Chris “do you know that he’s making a joke? Or do you just not think it’s funny?” Chris responded “I know he’s making a pun, I just don’t react because I don’t want to encourage him.” I said “This just encourages me more because now I feel challenged!”

 I wish I had tried harder because now I’ll never get that reaction.

 Kindness

 My last conversation with Chris was Thursday December 1st at RubyConf in Houston. We all went to dinner at a Ramen shop. I find British English to be extremely adorable, so any time I hear fun British phrases in the news I always ask my British coworkers about it. The latest one was “Wonky Veg” so I asked Chris if he’d been buying any at the store. He said no, but that one of his favorite things to do was find weird things at the local supermarket, take photos of it, then share with his coworkers. He flipped through photos on his phone, showing me pics of him shopping with his daughter. Some of the products he showed me were quite funny and we both had a good laugh.

 Memory

 I feel honored to have had the opportunity to work with Chris.

 I feel grateful for the time that we had together.

 I feel angry that I can’t learn more from him.

 I feel sad that he is gone from my life.

 Chris was an important part of the community, his family, and his country. I will never forget the time I spent with Chris, a Giant.

 read more »

 Cross Platform Machine Code

 2022-06-12 @ 12:54

 I hate writing if statements.

 I’ve been working on a couple different assemblers for Ruby. Fisk is a pure Ruby x86 assembler. You can use it to generate bytes that can be executed on x86 machines. AArch64 is a pure Ruby ARM64 assembler. You can use it to generate bytes that can be executed on ARM64 machines.

 Both of these libraries just generate bytes that can be interpreted by their respective processors. Unfortunately you can’t just generate bytes and expect the CPU to execute them. You first need to put the bytes in executable memory before you can hand them off to the CPU for execution. Executable memory is basically the same thing regardless of CPU architecture, so I decided to make a library called JITBuffer that encapsulates executable memory manipulation.

 To use the JITBuffer, you write platform specific bytes to the buffer, then give the buffer to the CPU for execution. Here is an example on the ARM64 platform:

 require "aarch64"
 require "jit_buffer"
 require "fiddle"
 
 asm = AArch64::Assembler.new
 
 # Make some instructions. These instructions simply
 # return the value 0xF00DCAFE
 asm.pretty do
 asm.movz x0, 0xCAFE
 asm.movk x0, 0xF00D, lsl(16)
 asm.ret
 end
 
 # Write the bytes to executable memory:
 buf = JITBuffer.new(4096)
 buf.writeable!
 asm.write_to buf
 buf.executable!
 
 # Point the CPU at the executable memory
 func = Fiddle::Function.new(buf.to_i, [], -Fiddle::TYPE_INT)
 p func.call.to_s(16) # => "f00dcafe"
 

 The example uses the AArch64 gem to assemble ARM64 specific bytes, the JITBuffer gem to allocate executable memory, and the Fiddle gem to point the CPU at the executable memory and run it.

 Tests are important I guess, so I thought it would be a good idea to write tests for the JITBuffer gem. My goal for the test is to ensure that it’s actually possible to execute the bytes in the buffer itself. I’m not a huge fan of stubs or mocks and I try to avoid them if possible, so I wanted to write a test that would actually execute the bytes in the buffer. I also want the test to be “cross platform” (where “cross platform” means “works on x86_64 and ARM64”).

 Writing a test like this would mean writing something like the following:

 def test_can_execute
 buf = JITBuffer.new(4096)
 
 platform = figure_out_what_platform_we_are_on()
 if platform == "arm64"
 # write arm64 specific bytes
 buf.write(...)
 else
 # write x86_64 specific bytes
 buf.write(...)
 end
 
 # Use fiddle to execute
 end
 

 As I said at the start though, I hate writing if statements, and I’d rather avoid it if possible. In addition, how do you reliably figure out what platform you’re executing on? I really don’t want to figure that out. Not to mention, I just don’t think this code is cool.

 My test requirements:

 * No if statements
 * Self contained (I don’t want to shell out or use other libraries)
 * Must have pizzazz

 Since machine code is just bytes that the CPU interprets, it made me wonder “is there a set of bytes that execute both on an x86_64 CPU and an ARM64 CPU?” It turns out there are, and I want to walk through them here.

 x86_64 Instructions

 First lets look at the x86_64 instructions we’ll execute. Below is the assembly code (in Intel syntax):

 .start:
 mov rax, 0x2b ; put 0x2b in the rax register
 ret ; return from the function
 jmp start ; jump to .start
 

 This assembly code puts the value 0x2b in the rax register and returns from the current “C” function. I put “C” in quotes because we’re writing assembly code, but the assembly code is conforming to the C calling convention and we’ll treat it as if it’s a C function when we call it. The x86 C calling convention states that the value in the rax register is the “return value” of the C function. So we’ve created a function that returns 0x2b. At the end of the code there is a jmp instruction that jumps to the start of this sequence. However, since we return from the function before getting to the jump, the jump is never used (or is it?????)

 Machine code is just bytes, and here are the bytes for the above x86 machine code:

 0x48 0xC7 0xC0 0x2b 0x00 0x00 0x00 ; mov rax, 0x2b
 0xC3 ; ret
 0xEB 0xF6 ; jmp start
 

 x86 uses a “variable width” encoding, meaning that the number of bytes each instruction uses can vary. In this example, the mov instruction used 7 bytes, and the ret instruction used 1 byte. This means that the jmp instruction is the 9th byte, or offset 8.

 ARM64 Instructions

 Below are some ARM64 instructions we can execute:

 movz X11, 0x7b7 ; put 0x7b7 in the X11 register
 movz X0, 0x2b ; put 0x2b in the X0 register
 ret ; return from the function
 

 This machine code puts the value 0x7b7 in to the register X11. Then it puts the value 0x2b in the X0 register. The third instruction returns from the function. Again we are abiding by the C calling convention, but this time on the ARM64 platform. On the ARM64 platform, the value stored in X0 is the return value. So the above machine code will return the value 0x2b to the caller just like the x86_64 machine code did.

 Here are the bytes that represent the above ARM64 machine code:

 0xEB 0xF6 0x80 0xD2 ; movz X11, 0x7b7
 0x60 0x05 0x80 0xD2 ; movz X0, 0x2b
 0xC0 0x03 0x5F 0xD6 ; ret
 

 ARM64 uses fixed width instructions. All instructions on ARM64 are 32 bits wide.

 Cross Platform Machine Code

 Lets look at the byte blocks next to each other:

 ; x86_64 bytes
 0x48 0xC7 0xC0 0x2b 0x00 0x00 0x00 ; mov rax, 0x2b
 0xC3 ; ret
 0xEB 0xF6 ; jmp start
 
 ; ARM64 bytes
 0xEB 0xF6 0x80 0xD2 ; movz X11, 0x7b7
 0x60 0x05 0x80 0xD2 ; movz X0, 0x2b
 0xC0 0x03 0x5F 0xD6 ; ret
 

 Looking at the bytes, you’ll notice that the first two bytes of the ARM64 code (0xEB 0xF6) are exactly the same as the last two bytes of the x86_64 code. The first movz instruction in the ARM64 code was specially crafted as to have the same bytes as the last jmp instruction in the x86 code.

 If we combine these bytes, then tell the CPU to execute starting at a particular offset, then the interpretation of the bytes will change depending on the CPU, but the result is the same.

 Here are the bytes combined:

 0x48 0xC7 0xC0 0x2b 0x00 0x00 0x00 ; mov rax, 0x2b
 0xC3 ; ret
 start -> 0xEB 0xF6 0x80 0xD2 ; (jmp start, or movz X11, 0x7b7)
 0x60 0x05 0x80 0xD2 ; movz X0, 0x2b
 0xC0 0x03 0x5F 0xD6 ; ret
 

 Regardless of platform, we’ll tell the CPU to start executing from offset 8 in the byte buffer. If it’s an x86 CPU, it will interpret the bytes as a jump, execute the top bytes, return at the ret, and ignore the rest of the bytes in the buffer (as they are never reached). If it’s an ARM64 machine, then it will interpret the bytes as “put 0x7b7 in the X11 register” and continue, never seeing the x86 specific bytes at the start of the buffer.

 Both x86_64 and ARM64 platforms will return the same value 0x2b.

 Now we can write a test without if statements like this:

 def test_execute
 # Cross platform bytes
 bytes = [0x48, 0xc7, 0xc0, 0x2b, 0x00, 0x00, 0x00, # x86_64 mov rax, 0x2b
 0xc3, # x86_64 ret
 0xeb, 0xf6, # x86 jmp
 0x80, 0xd2, # ARM movz X11, 0x7b7
 0x60, 0x05, 0x80, 0xd2, # ARM movz X0, #0x2b
 0xc0, 0x03, 0x5f, 0xd6] # ARM ret
 
 # Write them to the buffer
 jit = JITBuffer.new(4096)
 jit.writeable!
 jit.write bytes.pack("C*")
 jit.executable!
 
 # start at offset 8
 offset = 8
 func = Fiddle::Function.new(jit.to_i + offset, [], Fiddle::TYPE_INT)
 
 # Check the return value
 assert_equal 0x2b, func.call
 end
 

 So simple!

 So cool!

 Tons of pizzazz!

 This test will execute machine code on both x86_64 as well as ARM64 and the machine code will return the same value. Not to mention, there’s no way RuboCop or Flay could possibly complain about this code. ð¤£

 I hope this inspires you to try writing cross platform machine code. This code only supports 2 platforms, but it does make me wonder how far we could stretch this and how many platforms we could support.

 Anyway, hope you have a good day!

 read more »

 Homebrew, Rosetta, and Ruby

 2022-01-07 @ 12:39

 Hi everyone! I finally upgraded to an M1. It’s really really great, but the main problem is that some projects I work on like TenderJIT and YJIT only really work on x86_64 and these new M1 machines use ARM chips. Fortunately we can run x86_64 software via Rosetta, so we can still do development work on x86 specific software.

 I’ve seen some solutions for setting up a dev environment that uses Rosetta, but I’d like to share what I did.

 Installing Homebrew

 I think most people recommend that you install two different versions of Homebrew, one that targets ARM, and the other that targets x86.

 So far, I’ve found this to be the best solution, so I went with it. Just do the normal Homebrew installation for ARM like this:

 $ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
 

 Then run the installer again under Rosetta like this:

 $ arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
 

 After doing this, I ended up with a Homebrew installation in /opt/homebrew (the ARM version), and another installation in /usr/local (the x86 version).

 Configuring your Terminal

 I read many places on the web that recommend you duplicate terminal, then rename it and modify the renamed version to run under Rosetta.

 I really didn’t like this solution. The problem for me is that I’d end up with two different terminal icons when doing cmd-tab, and I really can’t be bothered to read whether the terminal is the Rosetta one or not. It makes switching to the right terminal take way too long.

 Instead I decided to make my shell figure out what architecture I’m using, then update $PATH depending on whether I’m using x86 or ARM. To accomplish this, I installed Fish (I use Fish shell) in both the x86 and ARM installations of Homebrew:

 $ /opt/homebrew/bin/brew install fish
 $ arch -x86_64 /usr/local/bin/brew install fish
 

 If you’re not using Fish you don’t need to do this step. ð

 Next is the “interesting” part. In my config.fish, I added this:

 if test (arch) = "i386"
 set HOMEBREW_PREFIX /usr/local
 else
 set HOMEBREW_PREFIX /opt/homebrew
 end
 
 # Add the Homebrew prefix to $PATH. -m flag ensures it's at the beginning
 # of the path since the path might already be in $PATH (just not at the start)
 fish_add_path -m --path $HOMEBREW_PREFIX/bin
 
 alias intel 'arch -x86_64 /usr/local/bin/fish'
 

 The arch command will tell you which architecture you’re on. If I’m on i386, set the Homebrew prefix to /usr/local, otherwise set it to /opt/homebrew. Then use fish_add_path to prepend the Homebrew prefix to my $PATH environment variable. The -m switch moves the path to the front if $PATH already contained the path I’m trying to add.

 Finally I added an alias intel that just starts a new shell but under Rosetta. So my default workflow is to open a terminal under ARM, and if I need to work on an intel project, just run intel.

 How do I know my current architecture?

 The arch command will tell you the current architecture, but I don’t want to run that every time I want to verify my current architecture. My solution was to add an emoji to my prompt. I don’t like adding more text to my prompt, but this seems important enough to warrant the addition.

 My fish_prompt function looks like this:

 function fish_prompt --description 'Write out the prompt'
 if not set -q __fish_prompt_normal
 set -g __fish_prompt_normal (set_color normal)
 end
 
 if not set -q __fish_prompt_cwd
 set -g __fish_prompt_cwd (set_color $fish_color_cwd)
 end
 
 if test (arch) = "i386"
 set emote ð§
 else
 set emote ðª
 end
 
 echo -n -s "[$USER" @ (prompt_hostname) $emote ' ' "$__fish_prompt_cwd" (prompt_pwd) (__fish_vcs_prompt) "$__fish_prompt_normal" ']$ '
 end
 

 If I’m on ARM, the prompt will have an ðª emoji, and if I’m on x86, the prompt will have a ð§ emoji.

 Just to give an example, here is a sample session in my terminal:

 Last login: Fri Jan 7 12:37:59 on ttys001
 Welcome to fish, the friendly interactive shell
 [aaron@tc-lan-adapterðª ~]$ which brew
 /opt/homebrew/bin/brew
 [aaron@tc-lan-adapterðª ~]$ arch
 arm64
 [aaron@tc-lan-adapterðª ~]$ intel
 Welcome to fish, the friendly interactive shell
 [aaron@tc-lan-adapterð§ ~]$ which brew
 /usr/local/bin/brew
 [aaron@tc-lan-adapterð§ ~]$ arch
 i386
 [aaron@tc-lan-adapterð§ ~]$ exit
 [aaron@tc-lan-adapterðª ~]$ arch
 arm64
 [aaron@tc-lan-adapterðª ~]$ which brew
 /opt/homebrew/bin/brew
 [aaron@tc-lan-adapterðª ~]$
 

 Now I can easily switch back and forth between x86 and ARM and my prompt tells me which I’m using.

 Ruby with chruby

 My Ruby dev environment is still a work in progress. I use chruby for changing Ruby versions. The problem is that all Ruby versions live in the same directory. chruby doesn’t know the difference between ARM versions and x86 versions. So for now I’m adding the architecture name to the directory:

 [aaron@tc-lan-adapterðª ~]$ chruby
 ruby-3.0.2
 ruby-arm64
 ruby-i386
 ruby-trunk
 [aaron@tc-lan-adapterðª ~]$
 

 So I have to be careful which Ruby I switch to. I’ve filed a ticket on ruby-install, and I think we can make this nicer.

 Specifically I’d like to add a subfolder in ~/.rubies for each architecture, then point chruby at the right subfolder depending on my current architecture. Essentially the same trick I used for $PATH and Homebrew, but for pointing chruby at the right place given my current architecture.

 For now I just have to be careful though!

 One huge caveat for Fish users is that the current version of chruby-fish is broken such that changes to $PATH end up getting lost (see this issue).

 To work around that issue, I’m using @ioquatix’s fork of chruby-fish which can be found here. I just checked out that version of chruby-fish in my git project folder and added this to my config.fish:

 # Point Fish at our local checkout of chruby-fish
 set fish_function_path $fish_function_path $HOME/git/chruby-fish/share/fish/vendor_functions.d
 

 Conclusion

 Getting a dev environment up and running with Rosetta wasn’t too bad, but I think having the shell fix up $PATH is a better solution than having two copies of Terminal.app

 The scripts I presented here were all Fish specific, but I don’t think it should be too hard to translate them to whatever shell you use.

 Anyway, I hope you have a good weekend!

 read more »

 Publishing Gems With Your YubiKey

 2021-10-26 @ 15:17

 The recent compromise of ua-parser-js has put the security and trust of published packages at the top of my mind lately. In order to mitigate the risk of any Ruby Gems I manage from being hijacked, I enabled 2FA on my RubyGems.org account. This means that whenever I publish a Ruby Gem, I have to enter a one time passcode.

 I have to admit, I find this to be a pain. Whenever I do a release of Rails, I have to enter a passcode over and over again because you can only push one Gem at a time.

 Finally I’ve found a way to deal with this. I can maintain account security and also not be hassled with OTP codes again, thanks to my YubiKey.

 This is just a short post about how to set up your YubiKey as an authenticator for RubyGems.org, and how to publish Gems without getting an OTP prompt.

 Install ykman

 ykman is a command line utility for interacting with your YubiKey. I installed it on my Mac with Homebrew:

 $ brew install ykman
 

 Set up 2FA as usual

 If you already have 2FA enabled, you’ll have to temporarily disable it.

 Just go through the normal 2FA setup process and when you’re presented with a QR code, you’ll use the text key to configure your YubiKey.

 Just do:

 $ ykman oath accounts add -t -o TOTP rubygems.org:youremail@example.org 123456
 

 But use your email address and replace 123456 with the code you got from RubyGems.org. The -t flag will require you to touch the YubiKey when you want to generate an OTP.

 Generate an OTP

 You can now generate an OTP like this:

 $ ykman oath accounts code -s rubygems.org
 

 Publishing a Gem without OTP Prompts

 You can supply an OTP code to the gem interface via an environment variable or a command line argument.

 The environment variable version is like this:

 $ GEM_HOST_OTP_CODE=$(ykman oath accounts code -s rubygems.org) gem push cool-gem-0.0.0.gem
 

 The command line argument is like this:

 $ gem push cool-gem-0.0.0.gem --otp $(ykman oath accounts code -s rubygems.org)
 

 I’ve used the environment variable version, but not the command line argument though.

 Final Thoughts

 I also did this for NPM, but I haven’t tried pushing a package yet so I’ll see how that goes. I don’t really have any other thoughts except that everyone should enable 2FA so that we can prevent situations like ua-parser-js. I’m not particularly interested in installing someone’s Bitcoin miner on my machine, and I’m also not interested in being hassled because my package was hijacked.

 Everyone, please stay safe and enable 2FA!

 –Aaron

 <3 <3

 read more »

 Debugging an Assertion Error in Ruby

 2021-02-03 @ 17:13

 I hope nobody runs in to a problem where they need the information in this post, but in case you do, I hope this post is helpful. (I’m talking to you, future Aaron! lol)

 I committed a patch to Ruby that caused the tests to start failing. This was the patch:

 commit 1be84e53d76cff30ae371f0b397336dee934499d
 Author: Aaron Patterson <tenderlove@ruby-lang.org>
 Date: Mon Feb 1 10:42:13 2021 -0800
 
 Don't pin `val` passed in to `rb_define_const`.
 
 The caller should be responsible for holding a pinned reference (if they
 need that)
 
 diff --git a/variable.c b/variable.c
 index 92d7d11eab..ff4f7964a7 100644
 --- a/variable.c
 +++ b/variable.c
 @@ -3154,7 +3154,6 @@ rb_define_const(VALUE klass, const char *name, VALUE val)
 if (!rb_is_const_id(id)) {
 rb_warn("rb_define_const: invalid name `%s' for constant", name);
 }
 - rb_gc_register_mark_object(val);
 rb_const_set(klass, id, val);
 }
 

 This patch is supposed to allow objects passed in to rb_define_const to move. As the commit message says, the caller should be responsible for keeping the value pinned. At the time I committed the patch, I thought that most callers of the function were marking the value passed in (as val), so we were pinning objects that something else would already pin. In other words, this code was being wasteful by chewing up GC time by pinning objects that were already pinned.

 Unfortunately the CI started to error shortly after I committed this patch. Clearly the patch was related, but how?

 In this post I am going to walk through the debugging tricks I used to find the error.

 Reproduction

 I was able to reproduce the error on my Linux machine by running the same command CI ran. Unfortunately since this bug is related to GC, the error was intermittent. To reproduce it, I just ran the tests in a loop until the process crashed like this:

 $ while test $status -eq 0
 env RUBY_TESTOPTS='-q --tty=no' make -j16 -s check
 end
 

 Before running this loop though, I made sure to do ulimit -c unlimited so that I would get a core file when the process crashed.

 The Error

 After the process crashed, the top of the error looked like this:

 <OBJ_INFO:rb_ractor_confirm_belonging@./ractor_core.h:327> 0x000055be8657f180 [0 ] T_NONE 
 /home/aaron/git/ruby/lib/bundler/environment_preserver.rb:47: [BUG] id == 0 but not shareable
 ruby 3.1.0dev (2021-02-03T17:35:37Z master 6b4814083b) [x86_64-linux]
 
 

 The Ractor verification routines crashed the process because a T_NONE object is “not sharable”. In other words you can’t share an object of type T_NONE between Ractors. This makes sense because T_NONE objects are actually empty slots in the GC. If a Ractor, or any other Ruby code sees a T_NONE object, then it’s clearly an error. Only the GC internals should ever be dealing with this type.

 The top of the C backtrace looked like this:

 -- C level backtrace information -------------------------------------------
 /home/aaron/git/ruby/ruby(rb_print_backtrace+0x14) [0x55be856e9816] vm_dump.c:758
 /home/aaron/git/ruby/ruby(rb_vm_bugreport) vm_dump.c:1020
 /home/aaron/git/ruby/ruby(bug_report_end+0x0) [0x55be854e2a69] error.c:778
 /home/aaron/git/ruby/ruby(rb_bug_without_die) error.c:778
 /home/aaron/git/ruby/ruby(rb_bug+0x7d) [0x55be854e2bb0] error.c:786
 /home/aaron/git/ruby/ruby(rb_ractor_confirm_belonging+0x102) [0x55be856cf6e2] ./ractor_core.h:328
 /home/aaron/git/ruby/ruby(vm_exec_core+0x4ff3) [0x55be856b0003] vm.inc:2224
 /home/aaron/git/ruby/tool/lib/test/unit/parallel.rb(rb_vm_exec+0x886) [0x55be856c9946]
 /home/aaron/git/ruby/ruby(load_iseq_eval+0xbb) [0x55be8554f66b] load.c:594
 /home/aaron/git/ruby/ruby(require_internal+0x394) [0x55be8554e3e4] load.c:1065
 /home/aaron/git/ruby/ruby(rb_require_string+0x973c4) [0x55be8554d8a4] load.c:1142
 /home/aaron/git/ruby/ruby(rb_f_require) load.c:838
 /home/aaron/git/ruby/ruby(vm_call_cfunc_with_frame+0x11a) [0x55be856dd6fa] ./vm_insnhelper.c:2897
 /home/aaron/git/ruby/ruby(vm_call_method_each_type+0xaa) [0x55be856d4d3a] ./vm_insnhelper.c:3387
 /home/aaron/git/ruby/ruby(vm_call_alias+0x87) [0x55be856d68e7] ./vm_insnhelper.c:3037
 /home/aaron/git/ruby/ruby(vm_sendish+0x200) [0x55be856d08e0] ./vm_insnhelper.c:4498
 

 The function rb_ractor_confirm_belonging was the function raising an exception.

 Debugging the Core File with LLDB

 I usually use clang / lldb when debugging. I’ve added scripts to Ruby’s lldb tools that let me track down problems more easily, so I prefer it over gcc / gdb.

 First I inspected the backtrace in the corefile:

 (lldb) target create "./ruby" --core "core.456156"
 Core file '/home/aaron/git/ruby/core.456156' (x86_64) was loaded.
 (lldb) bt
 * thread #1, name = 'ruby', stop reason = signal SIGABRT
 * frame #0: 0x00007fdc5fc8918b libc.so.6`raise + 203
 frame #1: 0x00007fdc5fc68859 libc.so.6`abort + 299
 frame #2: 0x000056362ac38bc6 ruby`die at error.c:765:5
 frame #3: 0x000056362ac38bb5 ruby`rb_bug(fmt=<unavailable>) at error.c:788:5
 frame #4: 0x000056362ae256e2 ruby`rb_ractor_confirm_belonging(obj=<unavailable>) at ractor_core.h:328:13
 frame #5: 0x000056362ae06003 ruby`vm_exec_core(ec=<unavailable>, initial=<unavailable>) at vm.inc:2224:5
 frame #6: 0x000056362ae1f946 ruby`rb_vm_exec(ec=<unavailable>, mjit_enable_p=<unavailable>) at vm.c:0
 frame #7: 0x000056362aca566b ruby`load_iseq_eval(ec=0x000056362b176710, fname=0x000056362ce96660) at load.c:594:5
 frame #8: 0x000056362aca43e4 ruby`require_internal(ec=<unavailable>, fname=<unavailable>, exception=1) at load.c:1065:21
 frame #9: 0x000056362aca38a4 ruby`rb_f_require [inlined] rb_require_string(fname=0x00007fdc38033178) at load.c:1142:18
 frame #10: 0x000056362aca3880 ruby`rb_f_require(obj=<unavailable>, fname=0x00007fdc38033178) at load.c:838
 frame #11: 0x000056362ae336fa ruby`vm_call_cfunc_with_frame(ec=0x000056362b176710, reg_cfp=0x00007fdc5f958de0, calling=<unavailable>) at vm_insnhelper.c:2897:11
 frame #12: 0x000056362ae2ad3a ruby`vm_call_method_each_type(ec=0x000056362b176710, cfp=0x00007fdc5f958de0, calling=0x00007ffe3b552128) at vm_insnhelper.c:3387:16
 frame #13: 0x000056362ae2c8e7 ruby`vm_call_alias(ec=0x000056362b176710, cfp=0x00007fdc5f958de0, calling=0x00007ffe3b552128) at vm_insnhelper.c:3037:12
 

 It’s very similar to the backtrace in the crash report. The first thing that was interesting to me was frame 5 in vm_exec_core. vm_exec_core is the main loop for the YARV VM. This program was crashing when executing some kind of instruction in the virtual machine.

 (lldb) f 5
 frame #5: 0x000056362ae06003 ruby`vm_exec_core(ec=<unavailable>, initial=<unavailable>) at vm.inc:2224:5
 2221	 /* ### Instruction trailers. ### */
 2222	 CHECK_VM_STACK_OVERFLOW_FOR_INSN(VM_REG_CFP, INSN_ATTR(retn));
 2223	 CHECK_CANARY(leaf, INSN_ATTR(bin));
 -> 2224	 PUSH(val);
 2225	 if (leaf) ADD_PC(INSN_ATTR(width));
 2226	# undef INSN_ATTR
 2227	
 (lldb) 
 

 Checking frame 5, we can see that it’s crashing when we push a value on to the stack. The Ractor function checks the value of objects being pushed on the VM stack, and in this case we have an object that is a T_NONE. The question is where did this value come from?

 The crash happened in the file vm.inc, line 2224. This file is a generated file, so I can’t link to it, but I wanted to know which instruction was being executed, so I pulled up that file.

 Line 2224 happened to be inside the opt_send_without_block instruction. So something is calling a method, and the return value of the method is a T_NONE object.

 But what method is being called, and on what object?

 Finding the called method

 The value ec, or “Execution Context” contains information about the virtual machine at runtime. On the ec, we can find the cfp or “Control Frame Pointer” which is a data structure representing the current executing stack frame. In lldb, I could see that frame 7 had the ec available, so I went to that frame to look at the cfp:

 (lldb) f 7
 frame #7: 0x000056362aca566b ruby`load_iseq_eval(ec=0x000056362b176710, fname=0x000056362ce96660) at load.c:594:5
 591 	 rb_ast_dispose(ast);
 592 	 }
 593 	 rb_exec_event_hook_script_compiled(ec, iseq, Qnil);
 -> 594 	 rb_iseq_eval(iseq);
 595 	}
 596 	
 597 	static inline enum ruby_tag_type
 (lldb) p *ec->cfp
 (rb_control_frame_t) $1 = {
 pc = 0x000056362c095d58
 sp = 0x00007fdc5f859330
 iseq = 0x000056362ca051f0
 self = 0x000056362b1d92c0
 ep = 0x00007fdc5f859328
 block_code = 0x0000000000000000
 __bp__ = 0x00007fdc5f859330
 }
 

 The control frame pointer has a pointer to the iseq or “Instruction Sequence” that is currently being executed. It also has a pc or “Program Counter”, and the program counter usually points at the instruction that will be executed next (in other words, not the currently executing instruction). Of other interest, the iseq also has the source location that corresponds to those instructions.

 Getting the Source File

 If we examine the iseq structure, we can find the source location of the code that is currently being executed:

 (lldb) p ec->cfp->iseq->body->location
 (rb_iseq_location_t) $4 = {
 pathobj = 0x000056362ca06960
 base_label = 0x000056362ce95a30
 label = 0x000056362ce95a30
 first_lineno = 0x0000000000000051
 node_id = 137
 code_location = {
 beg_pos = (lineno = 40, column = 4)
 end_pos = (lineno = 50, column = 7)
 }
 }
 (lldb) command script import -r ~/git/ruby/misc/lldb_cruby.py
 lldb scripts for ruby has been installed.
 (lldb) rp 0x000056362ca06960
 bits [ ]
 T_STRING: [FROZEN] (const char [57]) $6 = "/home/aaron/git/ruby/lib/bundler/environment_preserver.rb"
 (lldb) 
 

 The location info clearly shows us that the instructions are on line 40. The pathobj member contains the file name, but it is stored as a Ruby string. To print out the string, I imported the lldb CRuby extensions, then used the rp command and gave it the address of the path object.

 From the output, we can see that it’s crashing in the “environment_preserver.rb” file inside of the instructions that are defined on line 40. We’re not crashing on line 40, but the instructions are defined there.

 Those instructions are this method:

 def replace_with_backup
 ENV.replace(backup) unless Gem.win_platform?
 
 # Fallback logic for Windows below to workaround
 # https://bugs.ruby-lang.org/issues/16798. Can be dropped once all
 # supported rubies include the fix for that.
 
 ENV.clear
 
 backup.each {|k, v| ENV[k] = v }
 end
 

 It’s still not clear which of these method calls is breaking. In this function we have some method call that is returning a T_NONE.

 Finding The Method Call

 To find the method call, I disassembled the instruction sequence and checked the program counter:

 (lldb) command script import -r misc/lldb_disasm.py
 lldb Ruby disasm installed.
 (lldb) rbdisasm ec->cfp->iseq
 PC IDX insn_name(operands) 
 0x56362c095c20 0000 opt_getinlinecache( 6, (struct iseq_inline_cache_entry *)0x56362c095ee0 )
 0x56362c095c38 0003 putobject( (VALUE)0x14 )
 0x56362c095c48 0005 getconstant( ID: 0x807b )
 0x56362c095c58 0007 opt_setinlinecache( (struct iseq_inline_cache_entry *)0x56362c095ee0 )
 0x56362c095c68 0009 opt_send_without_block( (struct rb_call_data *)0x56362c095f20 )
 0x56362c095c78 0011 branchif( 15 )
 0x56362c095c88 0013 opt_getinlinecache( 6, (struct iseq_inline_cache_entry *)0x56362c095ef0 )
 0x56362c095ca0 0016 putobject( (VALUE)0x14 )
 0x56362c095cb0 0018 getconstant( ID: 0x370b )
 0x56362c095cc0 0020 opt_setinlinecache( (struct iseq_inline_cache_entry *)0x56362c095ef0 )
 0x56362c095cd0 0022 putself
 0x56362c095cd8 0023 opt_send_without_block( (struct rb_call_data *)0x56362c095f30 )
 0x56362c095ce8 0025 opt_send_without_block( (struct rb_call_data *)0x56362c095f40 )
 0x56362c095cf8 0027 pop
 0x56362c095d00 0028 opt_getinlinecache( 6, (struct iseq_inline_cache_entry *)0x56362c095f00 )
 0x56362c095d18 0031 putobject( (VALUE)0x14 )
 0x56362c095d28 0033 getconstant( ID: 0x370b )
 0x56362c095d38 0035 opt_setinlinecache( (struct iseq_inline_cache_entry *)0x56362c095f00 )
 0x56362c095d48 0037 opt_send_without_block( (struct rb_call_data *)0x56362c095f50 )
 0x56362c095d58 0039 pop
 0x56362c095d60 0040 putself
 0x56362c095d68 0041 opt_send_without_block( (struct rb_call_data *)0x56362c095f60 )
 0x56362c095d78 0043 send( (struct rb_call_data *)0x56362c095f70, (rb_iseq_t *)0x56362ca05178 )
 0x56362c095d90 0046 leave
 (lldb) p ec->cfp->pc
 (const VALUE *) $9 = 0x000056362c095d58
 

 First I loaded the disassembly helper script. It provides the rbdisasm function. Then I used rbdisasm on the instruction sequence. This printed out the instructions in mostly human readable form. Printing the PC showed a value of 0x000056362c095d58. Looking at the PC list in the disassembly shows that 0x000056362c095d58 corresponds to a pop instruction. But the PC always points at the next instruction that will execute, not the currently executing instruction. The currently executing instruction is the one right before the PC. In this case we can see it is opt_send_without_block, which lines up with the information we discovered from vm.inc.

 This is the 3rd from last method call in the block. At 0041 there is another opt_send_without_block, and then at 0043 there is a generic send call.

 Looking at the Ruby code, from the bottom of the method, we see a call to backup. It’s not a local variable, so it must be a method call. The code calls each on that, and each takes a block. These must correspond to the opt_send_without_block and the send at the end of the instruction sequence. Our crash is happening just before these two, so it must be the call to ENV.clear.

 If we read the implementation of ENV.clear, we can see that it returns a global variable called envtbl:

 VALUE
 rb_env_clear(void)
 {
 VALUE keys;
 long i;
 
 keys = env_keys(TRUE);
 for (i=0; i<RARRAY_LEN(keys); i++) {
 VALUE key = RARRAY_AREF(keys, i);
 const char *nam = RSTRING_PTR(key);
 ruby_setenv(nam, 0);
 }
 RB_GC_GUARD(keys);
 return envtbl;
 }
 

 This object is allocated here:

 envtbl = rb_obj_alloc(rb_cObject);
 

 And then it calls rb_define_global_const to define the ENV constant as a global:

 /*
 * ENV is a Hash-like accessor for environment variables.
 *
 * See ENV (the class) for more details.
 */
 rb_define_global_const("ENV", envtbl);
 

 If we read rb_define_global_const we can see that it just calls rb_define_const:

 void
 rb_define_global_const(const char *name, VALUE val)
 {
 rb_define_const(rb_cObject, name, val);
 }
 

 Before my patch, any object passed to rb_define_const would be pinned. Once I removed the pinning, that allowed the ENV variable to move around even though it shouldn’t.

 I reverted that patch here, and then sent a pull request to make rb_gc_register_mark_object a little bit smarter here.

 Conclusion

 TBH I don’t know what to conclude this with. Debugging errors kind of sucks, but I hope that the LLDB scripts I wrote make it suck a little less. Hope you’re having a good day!!!

 read more »

 Counting Write Barrier Unprotected Objects

 2020-08-26 @ 15:29

 This is just a quick post mostly as a note to myself (because I forget the jq commands). Ruby objects that are not protected with a write barrier must be examined on every minor GC. That means that any objects in your system that live for a long time and don’t have write barrier protection will cause unnecessary overhead on every minor collection.

 Heap dumps will tell you which objects have a write barrier. In Rails apps I use a small script to get a dump of the heap after boot:

 require 'objspace'
 require 'config/environment'
 
 GC.start
 
 File.open("heap.dump", "wb") do |f|
 ObjectSpace.dump_all(output: f)
 end
 

 The heap.dump file will have a list of all of the objects in the heap.

 Here is an example of an object with a write barrier:

 {"address":"0x7fec1b2ff940", "type":"IMEMO", "class":"0x7fec1b2ffd50", "imemo_type":"ment", "references":["0x7fec1b314908", "0x7fec1b2ffcd8"], "memsize":48, "flags":{"wb_protected":true, "old":true, "uncollectible":true, "marked":true}}
 

 Here is an example of an object without a write barrier:

 {"address":"0x7fec1b2ff760", "type":"ICLASS", "class":"0x7fec1a8c0f60", "references":["0x7fec1a8c9250", "0x7fec1b2fefe0"], "memsize":40}
 

 Objects with a write barrier will have "wb_protected":true in their flags section.

 I like to use jq to process heap dumps. Here is a command to find all of the unprotected objects, group them by type, then count them up:

 $ jq 'select(.flags.wb_protected | not) | .type' heap.dump | sort | uniq -c | sort -n
 1 "MATCH"
 2 "ARRAY"
 5 "ROOT"
 9 "FILE"
 323 "MODULE"
 927 "ICLASS"
 1631 "DATA"
 

 All of the objects listed here will be examined on every minor GC. If my Rails app is spending a lot of time in minor GCs, this is a good place to look.

 Ruby 2.8 (or 3.0) will eliminate ICLASS from this list (here is the commit).

 read more »

 Guide to String Encoding in Ruby

 2020-01-13 @ 06:00

 Encoding issues don’t seem to happen frequently, but that is a blessing and a curse. It’s great not to fix them very frequently, but when you do need to fix them, lack of experience can leave you feeling lost.

 This post is meant to be a sort of guide about what to do when you encounter different types of encoding errors in Ruby. First we’ll cover what an encoding object is, then we’ll look at common encoding exceptions and how to fix them.

 What are String encodings in Ruby?

 In Ruby, strings are a combination of an array of bytes, and an encoding object. We can access the encoding object on the string by calling encoding on the string object.

 For example:

 >> x = 'Hello World'
 >> x.encoding
 => #<Encoding:UTF-8>
 

 In my environment, the default encoding object associated with a string us the “UTF-8” encoding object. A graph of the object relationship looks something like this:

 Changing a String’s Encoding

 We can change encoding by two different methods:

 * String#force_encoding
 * String#encode

 The force_encoding method will mutate the string object and only change which encoding object the string points to. It does nothing to the bytes of the string, it merely changes the encoding object associated with the string. Here we can see that the return value of encoding changes after we call the force_encode method:

 >> x = 'Hello World'
 >> x.encoding
 => #<Encoding:UTF-8>
 >> x.force_encoding "US-ASCII"
 => "Hello World"
 >> x.encoding
 => #<Encoding:US-ASCII>
 

 The encode method will create a new string based on the bytes of the old string and associate the encoding object with the new string.

 Here we can see that the encoding of x remains the same, and calling encode returns a new string y which is associated with the new encoding:

 >> x = 'Hello World'
 >> x.encoding
 => #<Encoding:UTF-8>
 >> y = x.encode("US-ASCII")
 >> x.encoding
 => #<Encoding:UTF-8>
 >> y.encoding
 => #<Encoding:US-ASCII>
 

 Here is a visualization of the difference:

 Calling force_encoding mutates the original string, where encode creates a new string with a different encoding. Translating a string from one encoding to another is probably the “normal” use of encodings. However, developers will rarely call the encode method because Ruby will typically handle any necessary translations automatically. It’s probably more common to call the force_encoding method, and that is because strings can be associated with the wrong encoding.

 Strings Can Have the Wrong Encoding

 Strings can be associated with the wrong encoding object, and that is the source of most if not all encoding related exceptions. Let’s look at an example:

 >> x = "Hello \x93\xfa\x96\x7b"
 >> x.encoding
 => #<Encoding:UTF-8>
 >> x.valid_encoding?
 => false
 

 In this case, Ruby associated the string "Hello \x93\xfa\x96\x7b" with the default encoding UTF-8. However, many of the bytes in the string are not valid Unicode characters. We can check if the string is associated with a valid encoding object by calling valid_encoding? method. The valid_encoding? method will scan all bytes to see if they are valid for that particular encoding object.

 So how do we fix this? The answer depends on the situation. We need to think about where the data came from and where the data is going. Let’s say we’ll display this string on a webpage, but we do not know the correct encoding for the string. In that case we probably want to make sure the string is valid UTF-8, but since we don’t know the correct encoding for the string, our only choice is to remove the bad bytes from the string.

 We can remove the unknown bytes by using the scrub method:

 >> x = "Hello \x93\xfa\x96\x7b"
 >> x.valid_encoding?
 => false
 >> y = x.scrub
 >> y
 => "Hello ï¿½ï¿½ï¿½{"
 >> y.encoding
 => #<Encoding:UTF-8>
 >> y.valid_encoding?
 => true
 

 The scrub method will return a new string associated with the encoding but with all of the invalid bytes replaced by a replacement character, the diamond question mark thing.

 What if we do know the encoding of the source string? Actually the example above is using a string that’s encoding using Shift JIS. Let’s say we know the encoding, and we want to display the string on a webpage. In that case we tag the string by using force_encoding, and transcode to UTF-8:

 >> x = "Hello \x93\xfa\x96\x7b"
 >> x.force_encoding "Shift_JIS"
 => "Hello \x{93FA}\x{967B}"
 >> x.valid_encoding?
 => true
 >> x.encode "UTF-8" # display as UTF-8
 => "Hello æ¥æ¬"
 

 The most important thing to think about when dealing with encoding issues is “where did this data come from?” and “what will we do with this data?” Answering those two questions will drive all decisions about which encoding to use with which string.

 Encoding Depends on the Context

 Before we look at some common errors and their remediation, let’s look at one more example of the encoding context dependency. In this example, we’ll use some user input as a cache key, but we’ll also display the user input on a webpage. We’re going to use our source data (the user input) in two places: as a cache key, and something to display on a web page.

 Here’s the code:

 require "digest/md5"
 require "cgi"
 
 # Make a checksum
 def make_checksum string
 Digest::MD5.hexdigest string
 end
 
 # Not good HTML escaping (don't use this)
 # Returns a string with UTF-8 compatible encoding for display on a webpage
 def display_on_web string
 string.gsub(/>/, ">")
 end
 
 # User input from an unknown source
 x = "Hello \x93\xfa\x96\x7b"
 p ENCODING: x.encoding
 p VALID_ENCODING: x.valid_encoding?
 
 p display_on_web x
 p make_checksum x
 

 If we run this code, we’ll get an exception:

 $ ruby thing.rb
 {:ENCODING=>#<Encoding:UTF-8>}
 {:VALID_ENCODING=>false}
 Traceback (most recent call last):
 2: from thing.rb:20:in `<main>'
 1: from thing.rb:12:in `display_on_web'
 thing.rb:12:in `gsub': invalid byte sequence in UTF-8 (ArgumentError)
 

 The problem is that we have a string of unknown input with bytes that are not valid UTF-8 characters. We know we want to display this string on a UTF-8 encoded webpage, so lets scrub the string:

 require "digest/md5"
 require "cgi"
 
 # Make a checksum
 def make_checksum string
 Digest::MD5.hexdigest string
 end
 
 # Not good HTML escaping (don't use this)
 # Returns a string with UTF-8 compatible encoding for display on a webpage
 def display_on_web string
 string.gsub(/>/, ">")
 end
 
 # User input from an unknown source
 x = "Hello \x93\xfa\x96\x7b".scrub
 p ENCODING: x.encoding
 p VALID_ENCODING: x.valid_encoding?
 
 p display_on_web x
 p make_checksum x
 

 Now when we run the program, the output is like this:

 $ ruby thing.rb
 {:ENCODING=>#<Encoding:UTF-8>}
 {:VALID_ENCODING=>true}
 "Hello ï¿½ï¿½ï¿½{"
 "4dab6f63b4d3ae3279345c9df31091eb"
 

 Great! We’ve build some HTML and generated a checksum. Unfortunately there is a bug in this code (of course the mere fact that we’ve written code means there’s a bug! lol) Let’s introduce a second user input string with slightly different bytes than the first input string:

 require "digest/md5"
 require "cgi"
 
 # Make a checksum
 def make_checksum string
 Digest::MD5.hexdigest string
 end
 
 # Not good HTML escaping (don't use this)
 # Returns a string with UTF-8 compatible encoding for display on a webpage
 def display_on_web string
 string.gsub(/>/, ">")
 end
 
 # User input from an unknown source
 x = "Hello \x93\xfa\x96\x7b".scrub
 p ENCODING: x.encoding
 p VALID_ENCODING: x.valid_encoding?
 
 p display_on_web x
 p make_checksum x
 
 # Second user input from an unknown source with slightly different bytes
 y = "Hello \x94\xfa\x97\x7b".scrub
 p ENCODING: y.encoding
 p VALID_ENCODING: y.valid_encoding?
 
 p display_on_web y
 p make_checksum y
 

 Here is the output from the program:

 $ ruby thing.rb
 {:ENCODING=>#<Encoding:UTF-8>}
 {:VALID_ENCODING=>true}
 "Hello ï¿½ï¿½ï¿½{"
 "4dab6f63b4d3ae3279345c9df31091eb"
 {:ENCODING=>#<Encoding:UTF-8>}
 {:VALID_ENCODING=>true}
 "Hello ï¿½ï¿½ï¿½{"
 "4dab6f63b4d3ae3279345c9df31091eb"
 

 The program works in the sense that there is no exception. But both user input strings have the same checksum despite the fact that the original strings clearly have different bytes! So what is the correct fix for this program? Again, we need to think about the source of the data (where did it come from), as well as what we will do with it (where it is going). In this case we have one source, from a user, and the user provided us with no encoding information. In other words, the encoding information of the source data is unknown, so we can only treat it as a sequence of bytes. We have two output cases, one is a UTF-8 HTML the other output is the input to our checksum function. The HTML requires that our string be UTF-8 so making the string valid UTF-8, in other words “scrubbing” it, before displaying makes sense. However, our checksum function requires seeing the original bytes of the string. Since the checksum is only concerned with the bytes in the string, any encoding including an invalid encoding will work. It’s nice to make sure all our strings have valid encodings though, so we’ll fix this example such that everything has a valid encoding.

 require "digest/md5"
 require "cgi"
 
 # Make a checksum
 def make_checksum string
 Digest::MD5.hexdigest string
 end
 
 # Not good HTML escaping (don't use this)
 # Returns a string with UTF-8 compatible encoding for display on a webpage
 def display_on_web string
 string.gsub(/>/, ">")
 end
 
 # User input from an unknown source
 x = "Hello \x93\xfa\x96\x7b".b
 p ENCODING: x.encoding
 p VALID_ENCODING: x.valid_encoding?
 
 p display_on_web x.encode("UTF-8", undef: :replace)
 p make_checksum x
 
 # Second user input from an unknown source with slightly different bytes
 y = "Hello \x94\xfa\x97\x7b".b
 p ENCODING: y.encoding
 p VALID_ENCODING: y.valid_encoding?
 
 p display_on_web y.encode("UTF-8", undef: :replace)
 p make_checksum y
 

 Here is the output of the program:

 $ ruby thing.rb
 {:ENCODING=>#<Encoding:ASCII-8BIT>}
 {:VALID_ENCODING=>true}
 "Hello ï¿½ï¿½ï¿½{"
 "96cf6db2750fd4d2488fac57d8e4d45a"
 {:ENCODING=>#<Encoding:ASCII-8BIT>}
 {:VALID_ENCODING=>true}
 "Hello ï¿½ï¿½ï¿½{"
 "b92854c0db4f2c2c20eff349a9a8e3a0"
 

 To fix our program, we’ve changed a couple things. First we tagged the string of unknown encoding as “binary” by using the .b method. The .b method returns a new string that is associated with the ASCII-8BIT encoding. The name ASCII-8BIT is somewhat confusing because it has the word “ASCII” in it. It’s better to think of this encoding as either “unknown” or “binary data”. Unknown meaning we have some data that may have a valid encoding, but we don’t know what it is. Or binary data, as in the bytes read from a JPEG file or some such binary format. Anyway, we pass the binary string in to the checksum function because the checksum only cares about the bytes in the string, not about the encoding.

 The second change we made is to call encode with the encoding we want (UTF-8) along with undef: :replace meaning that any time Ruby encounters bytes it doesn’t know how to convert to the target encoding, it will replace them with the replacement character (the diamond question thing).

 SIDE NOTE: This is probably not important, but it is fun! We can specify what Ruby uses for replacing unknown bytes. Here’s an example:

 >> x = "Hello \x94\xfa\x97\x7b".b
 >> x.encoding
 => #<Encoding:ASCII-8BIT>
 >> x.encode("UTF-8", undef: :replace, replace: "Aaron")
 => "Hello AaronAaronAaron{"
 >> x.encode("UTF-8", undef: :replace, replace: "ð¤£")
 => "Hello ð¤£ð¤£ð¤£{"
 >> [_.encoding, _.valid_encoding?]
 => [#<Encoding:UTF-8>, true]
 

 Now lets take a look at some common encoding errors in Ruby and what to do about them.

 Encoding::InvalidByteSequenceError

 This exception occurs when Ruby needs to examine the bytes in a string and the bytes do not match the encoding. Here is an example of this exception:

 >> x = "Hello \x93\xfa\x96\x7b"
 >> x.encode "UTF-16"
 Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
 2: from (irb):4
 1: from (irb):4:in `encode'
 Encoding::InvalidByteSequenceError ("\x93" on UTF-8)
 >> x.encoding
 => #<Encoding:UTF-8>
 >> x.valid_encoding?
 => false
 

 The string x contains bytes that aren’t valid UTF-8, yet it is associated with the UTF-8 encoding object. When we try to convert x to UTF-16, an exception occurs.

 How to fix Encoding::InvalidByteSequenceError

 Like most encoding issues, our string x is tagged with the wrong encoding. The way to fix this issue is to tag the string with the correct encoding. But what is the correct encoding? To figure out the correct encoding, you need to know where the string came from. For example if the string came from a Mime attachment, the Mime attachment should specify the encoding (or the RFC will tell you).

 In this case, the string is a valid Shift JIS string, but I know that because I looked up the bytes and manually entered them. So we’ll tag this as Shift JIS, and the exception goes away:

 >> x = "Hello \x93\xfa\x96\x7b"
 >> x.force_encoding "Shift_JIS"
 => "Hello \x{93FA}\x{967B}"
 >> x.encode "UTF-16"
 => "\uFEFFHello \u65E5\u672C"
 >> x.encoding
 => #<Encoding:Shift_JIS>
 >> x.valid_encoding?
 => true
 

 If you don’t know the source of the string, an alternative solution is to tag as UTF-8 and then scrub the bytes:

 >> x = "Hello \x93\xfa\x96\x7b"
 >> x.force_encoding "UTF-8"
 => "Hello \x93\xFA\x96{"
 >> x.scrub!
 => "Hello ï¿½ï¿½ï¿½{"
 >> x.encode "UTF-16"
 => "\uFEFFHello \uFFFD\uFFFD\uFFFD{"
 >> x.encoding
 => #<Encoding:UTF-8>
 >> x.valid_encoding?
 => true
 

 Of course this works, but it means that you’ve lost data. The best solution is to figure out what the encoding of the string should be depending on its source and tag it with the correct encoding.

 Encoding::UndefinedConversionError

 This exception occurs when a string of one encoding can’t be converted to another encoding.

 Here is an example:

 >> x = "å\u2160"
 >> x
 => "åâ "
 >> x.encoding
 => #<Encoding:UTF-8>
 >> x.valid_encoding?
 => true
 >> x.encode "Shift_JIS"
 Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
 2: from (irb):23
 1: from (irb):23:in `encode'
 Encoding::UndefinedConversionError (U+2160 from UTF-8 to Shift_JIS)
 

 In this example, we have two characters: “å”, and the Roman numeral 1 (“â ”). Unicode Roman numeral 1 cannot be converted to Shift JIS because there are two codepoints that represent that character in Shift JIS. This means the conversion is ambiguous, so Ruby will raise an exception.

 How to fix Encoding::UndefinedConversionError

 Our original string is correctly tagged as UTF-8, but we need to convert to Shift JIS. In this case we’ll use a replacement character when converting to Shift JIS:

 >> x = "å\u2160"
 >> y = x.encode("Shift_JIS", undef: :replace)
 >> y
 => "\x{8E6C}?"
 >> y.encoding
 => #<Encoding:Shift_JIS>
 >> y.valid_encoding?
 => true
 >> y.encode "UTF-8"
 => "å?"
 

 We were able to convert to Shift JIS, but we did lose some data.

 ArgumentError

 When a string contains invalid bytes, sometimes Ruby will raise an ArgumentError exception:

 >> x = "Hello \x93\xfa\x96\x7b"
 >> x.downcase
 Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
 2: from (irb):34
 1: from (irb):34:in `downcase'
 ArgumentError (input string invalid)
 >> x.gsub(/ello/, "i")
 Traceback (most recent call last):
 6: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
 3: from (irb):34
 2: from (irb):35:in `rescue in irb_binding'
 1: from (irb):35:in `gsub'
 ArgumentError (invalid byte sequence in UTF-8)
 

 Again we use our incorrectly tagged Shift JIS string. Calling downcase or gsub both result in an ArgumentError. I personally think these exceptions are not great. We didn’t pass anything to downcase, so why is it an ArgumentError? There is nothing wrong with the arguments we passed to gsub, so why is it an ArgumentError? Why does one say “input string invalid” where the other gives us a slightly more helpful exception of “invalid byte sequence in UTF-8”? I think these should both result in Encoding::InvalidByteSequenceError exceptions, as it’s a problem with the encoding, not the arguments.

 Regardless, these errors both stem from the fact that the Shift JIS string is incorrectly tagged as UTF-8.

 Fixing ArgumentError

 Fixing this issue is just like fixing Encoding::InvalidByteSequenceError. We need to figure out the correct encoding of the source string, then tag the source string with that encoding. If the encoding of the source string is truly unknown, scrub it.

 >> x = "Hello \x93\xfa\x96\x7b"
 >> x.force_encoding "Shift_JIS"
 => "Hello \x{93FA}\x{967B}"
 >> x.downcase
 => "hello \x{93FA}\x{967B}"
 >> x.gsub(/ello/, "i")
 => "Hi \x{93FA}\x{967B}"
 

 Encoding::CompatibilityError

 This exception occurs when we try to combine strings of two different encodings and those encodings are incompatible. For example:

 >> x = "å\u2160"
 >> y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
 >> [x.encoding, x.valid_encoding?]
 => [#<Encoding:UTF-8>, true]
 >> [y.encoding, y.valid_encoding?]
 => [#<Encoding:Shift_JIS>, true]
 >> x + y
 Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
 2: from (irb):50
 1: from (irb):50:in `+'
 Encoding::CompatibilityError (incompatible character encodings: UTF-8 and Shift_JIS)
 

 In this example we have a valid UTF-8 string and a valid Shift JIS string. However, these two encodings are not compatible, so we get an exception when combining.

 Fixing Encoding::CompatibilityError

 To fix this exception, we need to manually convert one string to a new string that has a compatible encoding. In the case above, we can choose whether we want the output string to be UTF-8 or Shift JIS, and then call encode on the appropriate string.

 In the case we want UTF-8 output, we can do this:

 >> x = "å"
 >> y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
 >> x + y.encode("UTF-8")
 => "åHello æ¥æ¬"
 >> [_.encoding, _.valid_encoding?]
 => [#<Encoding:UTF-8>, true]
 

 If we wanted Shift JIS, we could do this:

 >> x = "å"
 >> y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
 >> x.encode("Shift_JIS") + y
 => "\x{8E6C}Hello \x{93FA}\x{967B}"
 >> [_.encoding, _.valid_encoding?]
 => [#<Encoding:Shift_JIS>, true]
 

 Another possible solution is to scrub bytes and concatenate, but again that results in data loss.

 What is a compatible encoding?

 If there are incompatible encodings, there must be compatible encodings too (at least I would think that). Here is an example of compatible encodings:

 >> x = "Hello World!".force_encoding "US-ASCII"
 >> [x.encoding, x.valid_encoding?]
 => [#<Encoding:US-ASCII>, true]
 >> y = "ããã«ã¡ã¯"
 >> [y.encoding, y.valid_encoding?]
 => [#<Encoding:UTF-8>, true]
 >> y + x
 => "ããã«ã¡ã¯Hello World!"
 >> [_.encoding, _.valid_encoding?]
 => [#<Encoding:UTF-8>, true]
 >> x + y
 => "Hello World!ããã«ã¡ã¯"
 >> [_.encoding, _.valid_encoding?]
 => [#<Encoding:UTF-8>, true]
 

 The x string is encoded with “US ASCII” encoding and the y string UTF-8. US ASCII is fully compatible with UTF-8, so even though these two strings have different encoding, concatenation works fine.

 String literals may default to UTF-8, but some functions will return US ASCII encoded strings. For example:

 >> require "digest/md5"
 => true
 >> Digest::MD5.hexdigest("foo").encoding
 => #<Encoding:US-ASCII>
 

 A hexdigest will only ever contain ASCII characters, so the implementation tags the returned string as US-ASCII.

 Encoding Gotchas

 Let’s look at a couple encoding gotcha’s.

 Infectious Invalid Encodings

 When a string is incorrectly tagged, Ruby will typically only raise an exception when it needs to actually examine the bytes. Here is an example:

 >> x = "Hello \x93\xfa\x96\x7b"
 >> x.encoding
 => #<Encoding:UTF-8>
 >> x.valid_encoding?
 => false
 >> x + "ã»ã"
 => "Hello \x93\xFA\x96{ã»ã"
 >> y = _
 >> y
 => "Hello \x93\xFA\x96{ã»ã"
 >> [y.encoding, y.valid_encoding?]
 => [#<Encoding:UTF-8>, false]
 

 Again we have the incorrectly tagged Shift JIS string. We’re able to append a correctly tagged UTF-8 string and no exception is raised. Why is that? Ruby assumes that if both strings have the same encoding, there is no reason to validate the bytes in either string so it will just append them. That means we can have an incorrectly tagged string “infect” what would otherwise be correctly tagged UTF-8 strings. Say we have some code like this:

 def append string
 string + "ã»ã"
 end
 
 p append("ã»ã").valid_encoding? # => true
 p append("Hello \x93\xfa\x96\x7b").valid_encoding? # = false
 

 When debugging this code, we may be tempted to think the problem is in the append method. But actually the issue is with the caller. The caller is passing in incorrectly tagged strings, and unfortunately we might not get an exception until the return value of append is used somewhere far away.

 ASCII-8BIT is Special

 Sometimes ASCII-8BIT is considered to be a “compatible” encoding and sometimes it isn’t. Here is an example:

 >> x = "\x93\xfa\x96\x7b".b
 >> x.encoding
 => #<Encoding:ASCII-8BIT>
 >> y = "ã»ã"
 >> y + x
 Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
 2: from (irb):89
 1: from (irb):89:in `+'
 Encoding::CompatibilityError (incompatible character encodings: UTF-8 and ASCII-8BIT)
 

 Here we have a binary string stored in x. Maybe it came from a JPEG file or something (it didn’t, I just typed it in!) When we try to concatenate the binary string with the UTF-8 string, we get an exception. But this may actually be an exception we want! It doesn’t make sense to be concatenating some JPEG data with an actual string we want to view, so it’s good we got an exception here.

 Now here is the same code, but with the contents of x changed somewhat:

 >> x = "Hello World".b
 >> x.encoding
 => #<Encoding:ASCII-8BIT>
 >> y = "ã»ã"
 >> y + x
 => "ã»ãHello World"
 

 We have the same code with the same encodings at play. The only thing that changed is the actual contents of the x string.

 When Ruby concatenates ASCII-8BIT strings, it will examine the contents of that string. If all bytes in the string are ASCII characters, it will treat it as a US-ASCII string and consider it to be “compatible”. If the string contains non-ASCII characters, it will consider it to be incompatible.

 This means that if you had read some data from your JPEG, and that data happened to all be ASCII characters, you would not get an exception even though maybe you really wanted one.

 In my personal opinion, concatenating an ASCII-8BIT string with anything besides another ASCII-8BIT string should be an exception.

 Anyway, this is all I feel like writing today. I hope you have a good day, and remember to check your encodings!

 read more »

 My Career Goals

 2019-10-12 @ 12:40

 I was going to tweet about this, but then I thought I’d have to make a bunch of tweets, and writing a blurgh post just seemed easier. Plus I don’t really have any puns in this post, so I can’t tweet it!

 My Career Goals

 I think many people aren’t sure what they want to do in their career. When I first started programming, I wasn’t sure what I wanted to do with my career. But after years of experience, my career aspirations have become crystal clear. I would like my job to be:

 * Improving Ruby and Rails internals
 * Teaching people

 Improving Ruby and Rails internals

 I got my first job programming in 1999. At that time, I didn’t know I wanted to be a programmer, it was just a way for me to pay for school. It turned out that I was pretty good at programming, so I decided that would be my career. To be honest, at that time I didn’t really love programming. I just found that I was good at it, and I could make decent money. In 2005 I found Ruby and Rails and that’s when I actually learned that I love programming. I loved Ruby so much that I learned Japanese so I could read blog posts about Ruby. 14 years later, I can easily read those blog posts, but I don’t actually need them. Oops!

 The reason I want to work on Ruby and Rails internals is that I want the language and framework to be performant, stable, easy to use. I want Ruby and Rails to be a great choice for people to use in production. I want others to experience the same joy I felt writing Ruby, and I want to make sure there are business that will employ those people.

 Teaching People

 I love to teach people things I know. I also love learning new things. As I hack on language and framework internals, I try to take that knowledge an disseminate it to as many people as I can.

 Why?

 First, I don’t think people can feel the joy of programming in Ruby/Rails unless they know how to actually program with Ruby/Rails. So I’m happy to help new folks get in to the language and framework.

 Second, I realize I’m not going to be around forever, and I want to make sure that these technologies will outlive me. If these technologies are going to survive in to the future, people need to understand how they work. Simply put: it’s an insurance policy for the future.

 Third, it’s just fun.

 Summary

 My dream job is to hack Ruby/Rails internals and teach people everything I know. Doing it is fun for me, and it’s the best way I can use my skills to make a real impact on the world.

 The End.

 read more »

 ESP8266 and Plantower Particle Sensor

 2019-09-03 @ 08:20

 Since forest fires have started to become a normal thing in the PNW, I’ve gotten interested in monitoring the air quality in and around my house. I found some sensors that will measure PM2.5 which is a standard for measuring air quality. The sensor I’m using is a PMS5003, and you can see the data sheet for it here. I like this sensor because it supports UART, so I was able to hook it to an FTDI and read data directly from my computer. I wanted to log the data, so I hooked it up to a Raspberry PI. However, I decided I’d like to measure the air quality in my office, a second room in the house, and also outside. Buying a Raspberry Pi for every sensor I purchase seems a little unreasonable, so I investigated a different solution. I settled on the ESP8266 E-01. This part can connect to wifi, knows how to speak UART, and is powerful enough to program directly. My plan was to read data from the sensor, then broadcast the data via UDP and have a central Raspberry Pi collect the data and report on it. Unfortunately, this plan has taken me many months to execute, so I’m going to write here the stuff I wish I had known when getting started.

 Parts

 Here are the parts I used:

 * ESP8266
 * Plantower PMS5003
 * ESP8266 Breadboard Adapter
 * ESP8266 Programmer

 Wiring

 Basically I just hooked the TX / RX pins to the Plantower sensor and set the CHPD and RST pins to high.

 Challenges with the ESP8266

 Now I’m basically going to complain about this chip, and then I’ll post the code I used.

 The first issue I ran in to is that I’m not sure what to call this thing, so searching the internet became a challenge. It seems that “ESP8266” refers to the chip, but E-01 refers to the package? I’m still not actually sure. It seems there are several boards that have an ESP8266 mounted on them, but searching for ESP8266 with E01 seemed to work.

 The second issue is programming the chip. I prefer to use C when developing for embedded systems, but no matter how hard I tried, I could not get the native toolchain running on MacOS. Finally I gave up and just used the Arduino toolchain. Somehow, you can write programs for the ESP8266 in Arduino, but doing it in C seems impossible (on Mac anyway).

 Building a circuit to program the chip seems impossible. I found some schematics online for building a programmer, but I couldn’t get anything to work. Instead, I ended up buying a dedicated programmer, and it seems to work well.

 Power requirements are extremely finicky. The chip wants 3.3v and at times 400mA. If either of these criteria aren’t met, the chip won’t work. Sometimes the chip wouldn’t do anything. Sometimes it would start, but when it tried to do wifi it would just restart. I ended up connecting a dedicated power supply to get the right power requirements.

 The ESP8266 E-01 is not breadboard friendly. I ended up buying some breadboard adapters so I could prototype.

 CHPD and RST need to be pulled HIGH for the chip to boot. This got me for a long time. I was able to program the chip with the programmer, but as soon as I moved it to the breadboard, nothing worked. In order to get the chip to actually boot, both CHPD and RST need to be pulled high.

 The air quality sensor is 5v. This isn’t too much of a problem, just kind of annoying that I really really have to use two different voltages for this task.

 Picture

 Here is a picture of the breadboard setup I have now:

 The blue box on the right is the air quality sensor, in the middle on the breadboard is the ESP8266, and up top is the power supply.

 Code

 Here is the Arduino code I used:

 #include <ESP8266WiFi.h>
 #include <WiFiUdp.h>
 #include <ESP8266WiFiMulti.h>
 #include <base64.h>
 
 #ifndef STASSID
 #define STASSID "WifiAPName"
 #define STAPSK "WifiPassword"
 #endif
 
 const char* ssid = STASSID;
 const char* password = STAPSK;
 
 ESP8266WiFiMulti WiFiMulti;
 WiFiUDP udp;
 IPAddress broadcastIp(224, 0, 0, 1);
 
 byte inputString[32];
 int i = 0;
 int recordId = 0;
 
 void setup() {
 Serial.begin(9600);
 
 WiFi.mode(WIFI_STA);
 WiFiMulti.addAP(ssid, password);
 
 while (WiFiMulti.run() != WL_CONNECTED) {
 delay(500);
 }
 
 delay(500);
 }
 
 void loop() {
 while (Serial.available()) {
 inputString[i] = Serial.read();
 i++;
 if (i == 2) { // Check for start of packet
 if (!(inputString[0] == 0x42 && inputString[1] == 0x4d)) {
 i = 0;
 }
 }
 if (i == 32) {
 i = 0;
 String encoded = base64::encode(inputString, 32);
 udp.beginPacketMulticast(broadcastIp, 9000, WiFi.localIP());
 udp.print("[\"aq\",{\"mac\":\"");
 udp.print(WiFi.macAddress());
 udp.print("\",\"record_id\":");
 udp.print(recordId);
 udp.print(",\"record\":\"");
 udp.print(encoded);
 udp.print("\"}]");
 udp.endPacket();
 recordId++;
 }
 }
 }
 

 I haven’t added CRC checking in this code, but it seems to work fine. Basically it reads data from the AQ sensor, Base64 encodes the data, then broadcasts the info as JSON over UDP on my network.

 Here is the client code:

 require "socket"
 require "ipaddr"
 require "json"
 
 MULTICAST_ADDR = "224.0.0.1"
 BIND_ADDR = "0.0.0.0"
 PORT = 9000
 
 if_addr = Socket.getifaddrs.find { |s| s.addr.ipv4? && !s.addr.ipv4_loopback? }
 p if_addr.addr.ip_address
 
 socket = UDPSocket.new
 membership = IPAddr.new(MULTICAST_ADDR).hton + IPAddr.new(BIND_ADDR).hton
 
 socket.setsockopt(:IPPROTO_IP, :IP_ADD_MEMBERSHIP, membership)
 socket.setsockopt(:IPPROTO_IP, :IP_MULTICAST_TTL, 1)
 socket.setsockopt(:SOL_SOCKET, :SO_REUSEPORT, 1)
 
 socket.bind(BIND_ADDR, PORT)
 
 class Sample < Struct.new(:time,
 :pm1_0_standard, :pm2_5_standard, :pm10_standard,
 :pm1_0_env, :pm2_5_env,
 :concentration_unit,
 
 # These fields are "number of particles beyond N um
 # per 0.1L of air". These numbers are multiplied by
 # 10, so 03um == "number of particles beyond 0.3um
 # in 0.1L of air"
 :particle_03um, :particle_05um, :particle_10um,
 :particle_25um, :particle_50um, :particle_100um)
 end
 
 loop do
 m, _ = socket.recvfrom(2000)
 record = JSON.load(m)[1]
 
 data = record["record"].unpack("m0").first
 unpack = data.unpack('CCnn14')
 crc = 0x42 + 0x4d + 28 + data.bytes.drop(4).first(26).inject(:+)
 unless crc != unpack.last
 p Sample.new(Time.now.utc, *unpack.drop(3).first(12))
 end
 end
 

 This code just listens for incoming data and prints it out.

 I’ve posted the code here.

 Conclusion

 This is what I did over the long weekend! Since the AQ sensor only uses the RX and TX pins on the ESP8266, it means I’ve got at least two more GPIO pins left. Next I’ll add temperature and humidity sensor, then make something a bit more permanent.

 read more »

Archives

 Copyright Â© 2023 Tenderlovemaking.com All rights reserved.

Mastodon
Encoding issues don’t seem to happen frequently, but that is a blessing and a curse.
It’s great not to fix them very frequently, but when you do need to fix them, lack
of experience can leave you feeling lost.
This post is meant to be a sort of guide about what to do when you encounter different types
of encoding errors in Ruby.
First we’ll cover what an encoding object is, then we’ll look at common encoding exceptions
and how to fix them.
What are String encodings in Ruby?
In Ruby, strings are a combination of an array of bytes, and an encoding object.
We can access the encoding object on the string by calling encoding on the
string object.
For example:
 &gt;&gt; x = 'Hello World'
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
 
In my environment, the default encoding object associated with a string us the “UTF-8” encoding object.
A graph of the object relationship looks something like this:
 
Changing a String’s Encoding
We can change encoding by two different methods:
 String#force_encoding
 String#encode
The force_encoding method will mutate the string object and only change which encoding object the string points to.
It does nothing to the bytes of the string, it merely changes the encoding object associated with the string.
Here we can see that the return value of encoding changes after we call the force_encode method:
 &gt;&gt; x = 'Hello World'
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.force_encoding "US-ASCII"
=&gt; "Hello World"
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:US-ASCII&gt;
 
The encode method will create a new string based on the bytes of the old string and associate the encoding object with the new string.
Here we can see that the encoding of x remains the same, and
calling encode returns a new string y which is associated with the new encoding:
 &gt;&gt; x = 'Hello World'
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; y = x.encode("US-ASCII")
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; y.encoding
=&gt; #&lt;Encoding:US-ASCII&gt;
 
Here is a visualization of the difference:
 
Calling force_encoding mutates the original string, where encode creates a new string with a different encoding.
Translating a string from one encoding to another is probably the “normal” use of encodings.
However, developers will rarely call the encode method because Ruby will typically handle any necessary translations automatically.
It’s probably more common to call the force_encoding method, and that is because strings can be associated with the wrong encoding.
Strings Can Have the Wrong Encoding
Strings can be associated with the wrong encoding object, and that is the source of most if not all encoding related exceptions.
Let’s look at an example:
 &gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.valid_encoding?
=&gt; false
 
In this case, Ruby associated the string "Hello \x93\xfa\x96\x7b" with the default encoding UTF-8.
However, many of the bytes in the string are not valid Unicode characters.
We can check if the string is associated with a valid encoding object by calling valid_encoding? method.
The valid_encoding? method will scan all bytes to see if they are valid for that particular encoding object.
So how do we fix this?
The answer depends on the situation.
We need to think about where the data came from and where the data is going.
Let’s say we’ll display this string on a webpage, but we do not know the correct encoding for the string.
In that case we probably want to make sure the string is valid UTF-8, but since we don’t know the correct encoding for the string, our only choice is to remove the bad bytes from the string.
We can remove the unknown bytes by using the scrub method:
 &gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.valid_encoding?
=&gt; false
&gt;&gt; y = x.scrub
&gt;&gt; y
=&gt; "Hello ï¿½ï¿½ï¿½{"
&gt;&gt; y.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; y.valid_encoding?
=&gt; true
 
The scrub method will return a new string associated with the encoding but with all of the invalid bytes replaced by a replacement character, the diamond question mark thing.
What if we do know the encoding of the source string?
Actually the example above is using a string that’s encoding using Shift JIS.
Let’s say we know the encoding, and we want to display the string on a webpage.
In that case we tag the string by using force_encoding, and transcode to UTF-8:
 &gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.force_encoding "Shift_JIS"
=&gt; "Hello \x{93FA}\x{967B}"
&gt;&gt; x.valid_encoding?
=&gt; true
&gt;&gt; x.encode "UTF-8" # display as UTF-8
=&gt; "Hello æ¥æ¬"
 
The most important thing to think about when dealing with encoding issues is “where did this data come from?” and “what will we do with this data?”
Answering those two questions will drive all decisions about which encoding to use with which string.
Encoding Depends on the Context
Before we look at some common errors and their remediation, let’s look at one more example of the encoding context dependency.
In this example, we’ll use some user input as a cache key, but we’ll also display the user input on a webpage.
We’re going to use our source data (the user input) in two places: as a cache key, and something to display on a web page.
Here’s the code:
 require "digest/md5"
require "cgi"
def make_checksum string
 Digest::MD5.hexdigest string
end
def display_on_web string
 string.gsub(/&gt;/, "&amp;gt;")
end
x = "Hello \x93\xfa\x96\x7b"
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?
p display_on_web x
p make_checksum x
 
If we run this code, we’ll get an exception:
 $ ruby thing.rb
{:ENCODING=&gt;#&lt;Encoding:UTF-8&gt;}
{:VALID_ENCODING=&gt;false}
Traceback (most recent call last):
 2: from thing.rb:20:in `&lt;main&gt;'
 1: from thing.rb:12:in `display_on_web'
thing.rb:12:in `gsub': invalid byte sequence in UTF-8 (ArgumentError)
 
The problem is that we have a string of unknown input with bytes that are not valid UTF-8 characters.
We know we want to display this string on a UTF-8 encoded webpage, so lets scrub the string:
 require "digest/md5"
require "cgi"
def make_checksum string
 Digest::MD5.hexdigest string
end
def display_on_web string
 string.gsub(/&gt;/, "&amp;gt;")
end
x = "Hello \x93\xfa\x96\x7b".scrub
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?
p display_on_web x
p make_checksum x
 
Now when we run the program, the output is like this:
 $ ruby thing.rb
{:ENCODING=&gt;#&lt;Encoding:UTF-8&gt;}
{:VALID_ENCODING=&gt;true}
"Hello ï¿½ï¿½ï¿½{"
"4dab6f63b4d3ae3279345c9df31091eb"
 
Great! We’ve build some HTML and generated a checksum.
Unfortunately there is a bug in this code (of course the mere fact that we’ve written code means there’s a bug! lol)
Let’s introduce a second user input string with slightly different bytes than the first input string:
 require "digest/md5"
require "cgi"
def make_checksum string
 Digest::MD5.hexdigest string
end
def display_on_web string
 string.gsub(/&gt;/, "&amp;gt;")
end
x = "Hello \x93\xfa\x96\x7b".scrub
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?
p display_on_web x
p make_checksum x
y = "Hello \x94\xfa\x97\x7b".scrub
p ENCODING: y.encoding
p VALID_ENCODING: y.valid_encoding?
p display_on_web y
p make_checksum y
 
Here is the output from the program:
 $ ruby thing.rb
{:ENCODING=&gt;#&lt;Encoding:UTF-8&gt;}
{:VALID_ENCODING=&gt;true}
"Hello ï¿½ï¿½ï¿½{"
"4dab6f63b4d3ae3279345c9df31091eb"
{:ENCODING=&gt;#&lt;Encoding:UTF-8&gt;}
{:VALID_ENCODING=&gt;true}
"Hello ï¿½ï¿½ï¿½{"
"4dab6f63b4d3ae3279345c9df31091eb"
 
The program works in the sense that there is no exception.
But both user input strings have the same checksum despite the fact that the original strings clearly have different bytes!
So what is the correct fix for this program?
Again, we need to think about the source of the data (where did it come from), as well as what we will do with it (where it is going).
In this case we have one source, from a user, and the user provided us with no encoding information.
In other words, the encoding information of the source data is unknown, so we can only treat it as a sequence of bytes.
We have two output cases, one is a UTF-8 HTML the other output is the input to our checksum function.
The HTML requires that our string be UTF-8 so making the string valid UTF-8, in other words “scrubbing” it, before displaying makes sense.
However, our checksum function requires seeing the original bytes of the string.
Since the checksum is only concerned with the bytes in the string, any encoding including an invalid encoding will work.
It’s nice to make sure all our strings have valid encodings though, so we’ll fix this example such that everything has a valid encoding.
 require "digest/md5"
require "cgi"
def make_checksum string
 Digest::MD5.hexdigest string
end
def display_on_web string
 string.gsub(/&gt;/, "&amp;gt;")
end
x = "Hello \x93\xfa\x96\x7b".b
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?
p display_on_web x.encode("UTF-8", undef: :replace)
p make_checksum x
y = "Hello \x94\xfa\x97\x7b".b
p ENCODING: y.encoding
p VALID_ENCODING: y.valid_encoding?
p display_on_web y.encode("UTF-8", undef: :replace)
p make_checksum y
 
Here is the output of the program:
 $ ruby thing.rb
{:ENCODING=&gt;#&lt;Encoding:ASCII-8BIT&gt;}
{:VALID_ENCODING=&gt;true}
"Hello ï¿½ï¿½ï¿½{"
"96cf6db2750fd4d2488fac57d8e4d45a"
{:ENCODING=&gt;#&lt;Encoding:ASCII-8BIT&gt;}
{:VALID_ENCODING=&gt;true}
"Hello ï¿½ï¿½ï¿½{"
"b92854c0db4f2c2c20eff349a9a8e3a0"
 
To fix our program, we’ve changed a couple things.
First we tagged the string of unknown encoding as “binary” by using the .b method.
The .b method returns a new string that is associated with the ASCII-8BIT encoding.
The name ASCII-8BIT is somewhat confusing because it has the word “ASCII” in it.
It’s better to think of this encoding as either “unknown” or “binary data”.
Unknown meaning we have some data that may have a valid encoding, but we don’t know what it is.
Or binary data, as in the bytes read from a JPEG file or some such binary format.
Anyway, we pass the binary string in to the checksum function because the checksum only cares about the bytes in the string, not about the encoding.
The second change we made is to call encode with the encoding we want (UTF-8) along with undef: :replace meaning that any time Ruby encounters bytes it doesn’t know how to convert to the target encoding, it will replace them with the replacement character (the diamond question thing).
SIDE NOTE: This is probably not important, but it is fun!
We can specify what Ruby uses for replacing unknown bytes.
Here’s an example:
 &gt;&gt; x = "Hello \x94\xfa\x97\x7b".b
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:ASCII-8BIT&gt;
&gt;&gt; x.encode("UTF-8", undef: :replace, replace: "Aaron")
=&gt; "Hello AaronAaronAaron{"
&gt;&gt; x.encode("UTF-8", undef: :replace, replace: "ð¤£")
=&gt; "Hello ð¤£ð¤£ð¤£{"
&gt;&gt; [_.encoding, _.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
 
Now lets take a look at some common encoding errors in Ruby and what to do about them.
 Encoding::InvalidByteSequenceError 
This exception occurs when Ruby needs to examine the bytes in a string and the bytes do not match the encoding.
Here is an example of this exception:
 &gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.encode "UTF-16"
Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 2: from (irb):4
 1: from (irb):4:in `encode'
Encoding::InvalidByteSequenceError ("\x93" on UTF-8)
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.valid_encoding?
=&gt; false
 
The string x contains bytes that aren’t valid UTF-8, yet it is associated with the UTF-8 encoding object.
When we try to convert x to UTF-16, an exception occurs.
How to fix Encoding::InvalidByteSequenceError 
Like most encoding issues, our string x is tagged with the wrong encoding.
The way to fix this issue is to tag the string with the correct encoding.
But what is the correct encoding?
To figure out the correct encoding, you need to know where the string came from.
For example if the string came from a Mime attachment, the Mime attachment should specify the encoding (or the RFC will tell you).
In this case, the string is a valid Shift JIS string, but I know that because I looked up the bytes and manually entered them.
So we’ll tag this as Shift JIS, and the exception goes away:
 &gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.force_encoding "Shift_JIS"
=&gt; "Hello \x{93FA}\x{967B}"
&gt;&gt; x.encode "UTF-16"
=&gt; "\uFEFFHello \u65E5\u672C"
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:Shift_JIS&gt;
&gt;&gt; x.valid_encoding?
=&gt; true
 
If you don’t know the source of the string, an alternative solution is to tag as UTF-8 and then scrub the bytes:
 &gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.force_encoding "UTF-8"
=&gt; "Hello \x93\xFA\x96{"
&gt;&gt; x.scrub!
=&gt; "Hello ï¿½ï¿½ï¿½{"
&gt;&gt; x.encode "UTF-16"
=&gt; "\uFEFFHello \uFFFD\uFFFD\uFFFD{"
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.valid_encoding?
=&gt; true
 
Of course this works, but it means that you’ve lost data.
The best solution is to figure out what the encoding of the string should be depending on its source and tag it with the correct encoding.
 Encoding::UndefinedConversionError 
This exception occurs when a string of one encoding can’t be converted to another encoding.
Here is an example:
 &gt;&gt; x = "å\u2160"
&gt;&gt; x
=&gt; "åâ "
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.valid_encoding?
=&gt; true
&gt;&gt; x.encode "Shift_JIS"
Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 2: from (irb):23
 1: from (irb):23:in `encode'
Encoding::UndefinedConversionError (U+2160 from UTF-8 to Shift_JIS)
 
In this example, we have two characters: “å”, and the Roman numeral 1 (“â ”).
Unicode Roman numeral 1 cannot be converted to Shift JIS because there are two codepoints that represent that character in Shift JIS.
This means the conversion is ambiguous, so Ruby will raise an exception.
How to fix Encoding::UndefinedConversionError 
Our original string is correctly tagged as UTF-8, but we need to convert to Shift JIS.
In this case we’ll use a replacement character when converting to Shift JIS:
 &gt;&gt; x = "å\u2160"
&gt;&gt; y = x.encode("Shift_JIS", undef: :replace)
&gt;&gt; y
=&gt; "\x{8E6C}?"
&gt;&gt; y.encoding
=&gt; #&lt;Encoding:Shift_JIS&gt;
&gt;&gt; y.valid_encoding?
=&gt; true
&gt;&gt; y.encode "UTF-8"
=&gt; "å?"
 
We were able to convert to Shift JIS, but we did lose some data.
 ArgumentError 
When a string contains invalid bytes, sometimes Ruby will raise an ArgumentError exception:
 &gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.downcase
Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 2: from (irb):34
 1: from (irb):34:in `downcase'
ArgumentError (input string invalid)
&gt;&gt; x.gsub(/ello/, "i")
Traceback (most recent call last):
 6: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 3: from (irb):34
 2: from (irb):35:in `rescue in irb_binding'
 1: from (irb):35:in `gsub'
ArgumentError (invalid byte sequence in UTF-8)
 
Again we use our incorrectly tagged Shift JIS string.
Calling downcase or gsub both result in an ArgumentError.
I personally think these exceptions are not great.
We didn’t pass anything to downcase, so why is it an ArgumentError?
There is nothing wrong with the arguments we passed to gsub, so why is it an ArgumentError?
Why does one say “input string invalid” where the other gives us a slightly more helpful exception of “invalid byte sequence in UTF-8”?
I think these should both result in Encoding::InvalidByteSequenceError exceptions, as it’s a problem with the encoding, not the arguments.
Regardless, these errors both stem from the fact that the Shift JIS string is incorrectly tagged as UTF-8.
Fixing ArgumentError 
Fixing this issue is just like fixing Encoding::InvalidByteSequenceError.
We need to figure out the correct encoding of the source string, then tag the source string with that encoding.
If the encoding of the source string is truly unknown, scrub it.
 &gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.force_encoding "Shift_JIS"
=&gt; "Hello \x{93FA}\x{967B}"
&gt;&gt; x.downcase
=&gt; "hello \x{93FA}\x{967B}"
&gt;&gt; x.gsub(/ello/, "i")
=&gt; "Hi \x{93FA}\x{967B}"
 
 Encoding::CompatibilityError 
This exception occurs when we try to combine strings of two different encodings and those encodings are incompatible.
For example:
 &gt;&gt; x = "å\u2160"
&gt;&gt; y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
&gt;&gt; [x.encoding, x.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
&gt;&gt; [y.encoding, y.valid_encoding?]
=&gt; [#&lt;Encoding:Shift_JIS&gt;, true]
&gt;&gt; x + y
Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 2: from (irb):50
 1: from (irb):50:in `+'
Encoding::CompatibilityError (incompatible character encodings: UTF-8 and Shift_JIS)
 
In this example we have a valid UTF-8 string and a valid Shift JIS string.
However, these two encodings are not compatible, so we get an exception when combining.
Fixing Encoding::CompatibilityError 
To fix this exception, we need to manually convert one string to a new string that has a compatible encoding.
In the case above, we can choose whether we want the output string to be UTF-8 or Shift JIS, and then call encode on the appropriate string.
In the case we want UTF-8 output, we can do this:
 &gt;&gt; x = "å"
&gt;&gt; y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
&gt;&gt; x + y.encode("UTF-8")
=&gt; "åHello æ¥æ¬"
&gt;&gt; [_.encoding, _.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
 
If we wanted Shift JIS, we could do this:
 &gt;&gt; x = "å"
&gt;&gt; y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
&gt;&gt; x.encode("Shift_JIS") + y
=&gt; "\x{8E6C}Hello \x{93FA}\x{967B}"
&gt;&gt; [_.encoding, _.valid_encoding?]
=&gt; [#&lt;Encoding:Shift_JIS&gt;, true]
 
Another possible solution is to scrub bytes and concatenate, but again that results in data loss.
What is a compatible encoding?
If there are incompatible encodings, there must be compatible encodings too (at least I would think that).
Here is an example of compatible encodings:
 &gt;&gt; x = "Hello World!".force_encoding "US-ASCII"
&gt;&gt; [x.encoding, x.valid_encoding?]
=&gt; [#&lt;Encoding:US-ASCII&gt;, true]
&gt;&gt; y = "ããã«ã¡ã¯"
&gt;&gt; [y.encoding, y.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
&gt;&gt; y + x
=&gt; "ããã«ã¡ã¯Hello World!"
&gt;&gt; [_.encoding, _.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
&gt;&gt; x + y
=&gt; "Hello World!ããã«ã¡ã¯"
&gt;&gt; [_.encoding, _.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, true]
 
The x string is encoded with “US ASCII” encoding and the y string UTF-8.
US ASCII is fully compatible with UTF-8, so even though these two strings have different encoding, concatenation works fine.
String literals may default to UTF-8, but some functions will return US ASCII encoded strings.
For example:
 &gt;&gt; require "digest/md5"
=&gt; true
&gt;&gt; Digest::MD5.hexdigest("foo").encoding
=&gt; #&lt;Encoding:US-ASCII&gt;
 
A hexdigest will only ever contain ASCII characters, so the implementation tags the returned string as US-ASCII.
Encoding Gotchas
Let’s look at a couple encoding gotcha’s.
Infectious Invalid Encodings
When a string is incorrectly tagged, Ruby will typically only raise an exception when it needs to actually examine the bytes.
Here is an example:
 &gt;&gt; x = "Hello \x93\xfa\x96\x7b"
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:UTF-8&gt;
&gt;&gt; x.valid_encoding?
=&gt; false
&gt;&gt; x + "ã»ã"
=&gt; "Hello \x93\xFA\x96{ã»ã"
&gt;&gt; y = _
&gt;&gt; y
=&gt; "Hello \x93\xFA\x96{ã»ã"
&gt;&gt; [y.encoding, y.valid_encoding?]
=&gt; [#&lt;Encoding:UTF-8&gt;, false]
 
Again we have the incorrectly tagged Shift JIS string.
We’re able to append a correctly tagged UTF-8 string and no exception is raised.
Why is that?
Ruby assumes that if both strings have the same encoding, there is no reason to validate the bytes in either string so it will just append them.
That means we can have an incorrectly tagged string “infect” what would otherwise be correctly tagged UTF-8 strings.
Say we have some code like this:
 def append string
 string + "ã»ã"
end
p append("ã»ã").valid_encoding? 
p append("Hello \x93\xfa\x96\x7b").valid_encoding? 
 
When debugging this code, we may be tempted to think the problem is in the append method.
But actually the issue is with the caller.
The caller is passing in incorrectly tagged strings, and unfortunately we might not get an exception until the return value of append is used somewhere far away.
ASCII-8BIT is Special
Sometimes ASCII-8BIT is considered to be a “compatible” encoding and sometimes it isn’t.
Here is an example:
 &gt;&gt; x = "\x93\xfa\x96\x7b".b
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:ASCII-8BIT&gt;
&gt;&gt; y = "ã»ã"
&gt;&gt; y + x
Traceback (most recent call last):
 5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `&lt;main&gt;'
 4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
 3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `&lt;top (required)&gt;'
 2: from (irb):89
 1: from (irb):89:in `+'
Encoding::CompatibilityError (incompatible character encodings: UTF-8 and ASCII-8BIT)
 
Here we have a binary string stored in x.
Maybe it came from a JPEG file or something (it didn’t, I just typed it in!)
When we try to concatenate the binary string with the UTF-8 string, we get an exception.
But this may actually be an exception we want!
It doesn’t make sense to be concatenating some JPEG data with an actual string we want to view, so it’s good we got an exception here.
Now here is the same code, but with the contents of x changed somewhat:
 &gt;&gt; x = "Hello World".b
&gt;&gt; x.encoding
=&gt; #&lt;Encoding:ASCII-8BIT&gt;
&gt;&gt; y = "ã»ã"
&gt;&gt; y + x
=&gt; "ã»ãHello World"
 
We have the same code with the same encodings at play.
The only thing that changed is the actual contents of the x string.
When Ruby concatenates ASCII-8BIT strings, it will examine the contents of that string.
If all bytes in the string are ASCII characters, it will treat it as a US-ASCII string and consider it to be “compatible”.
If the string contains non-ASCII characters, it will consider it to be incompatible.
This means that if you had read some data from your JPEG, and that data happened to all be ASCII characters, you would not get an exception even though maybe you really wanted one.
In my personal opinion, concatenating an ASCII-8BIT string with anything besides another ASCII-8BIT string should be an exception.
Anyway, this is all I feel like writing today. I hope you have a good day, and remember to check your encodings!
Vim, tmux, and Fish
http://tenderlovemaking.com/
https://masnun.com/
Abu Ashraf Masnun
2018-12-06
Umut Çağdaş Coşkun
<div><div>
	<p>	For building quick APIs in Python, I have mostly depended on <a href="http://polyglot.ninja/rest-api-best-practices-python-flask-tutorial/">Flask</a>. Recently I came across a new API framework for Python 3 named “API Star” which seemed really interesting to me for several reasons. Firstly the framework embraces modern Python features like type hints and asyncio. And then it goes ahead and uses these features to provide awesome development experience for us, the developers. We will get into those features soon but before we begin, I would like to thank Tom Christie for all the work he has put into Django REST Framework and now API Star.</p>
<p>Now back to API Star – I feel very productive in the framework. I can choose to write async codes based on asyncio or I can choose a traditional backend like WSGI. It comes with a command line tool – <code>apistar</code> to help us get things done faster. There’s (optional) support for both Django ORM and SQLAlchemy.  There’s a brilliant type system that enables us to define constraints on our input and output and from these, API Star can auto generate api schemas (and docs), provide validation and serialization feature and a lot more. Although API Star is heavily focused on building APIs, you can also build web applications on top of it fairly easily. All these might not make proper sense until we build something all by ourselves.</p>
<h3>Getting Started</h3>
<p>We will start by installing API Star. It would be a good idea to create a virtual environment for this exercise. If you don’t know how to create a virtualenv, don’t worry and go ahead.</p>
<pre>pip install apistar</pre>
<p>If you’re not using a virtual environment or the <code>pip</code> command for your Python 3 is called <code>pip3</code>, then please use <code>pip3 install apistar</code> instead.</p>
<p>Once we have the package installed, we should have access to the <code>apistar</code> command line tool. We can create a new project with it. Let’s create a new project in our current directory.</p>
<pre>apistar new .</pre>
<p>Now we should have two files created – <code>app.py</code> – which contains the main application and then <code>test.py</code> for our tests. Let’s examine our <code>app.py</code> file:</p>
<pre>from apistar import Include, Route
from apistar.frameworks.wsgi import WSGIApp as App
from apistar.handlers import docs_urls, static_urls
def welcome(name=None):
 if name is None:
 return {'message': 'Welcome to API Star!'}
 return {'message': 'Welcome to API Star, %s!' % name}
routes = [
 Route('/', 'GET', welcome),
 Include('/docs', docs_urls),
 Include('/static', static_urls)
]
app = App(routes=routes)
if __name__ == '__main__':
 app.main()</pre>
<p>Before we dive into the code, let’s run the app and see if it works. If we navigate to <code>http://127.0.0.1:8080/</code> we will get this following response:</p>
<pre>{"message": "Welcome to API Star!"}</pre>
<p>And if we navigate to: <code>http://127.0.0.1:8080/?name=masnun</code></p>
<pre>{"message": "Welcome to API Star, masnun!"}</pre>
<p>Similarly if we navigate to: <code>http://127.0.0.1:8080/docs/</code>, we will see auto generated docs for our API.</p>
<p>Now let’s look at the code. We have a <code>welcome</code> function that takes a parameter named <code>name</code> which has a default value of <code>None</code>. API Star is a smart api framework. It will try to find the <code>name</code> key in the url path or query string and pass it to our function. It also generates the API docs based on it. Pretty nice, no?</p>
<p>We then create a list of <code>Route</code> and <code>Include</code> instances and pass the list to the <code>App</code> instance. <code>Route</code> objects are used to define custom user routing. <code>Include</code> , as the name suggests, includes/embeds other routes under the path provided to it.</p>
<h3>Routing</h3>
<p>Routing is simple. When constructing the <code>App</code> instance, we need to pass a list as the <code>routes</code> argument. This list should comprise of <code>Route</code> or <code>Include</code> objects as we just saw above. For <code>Route</code>s, we pass a url path, http method name and the request handler callable (function or otherwise). For the <code>Include</code> instances, we pass a url path and a list of <code>Routes</code> instance.</p>
<h5>Path Parameters</h5>
<p>We can put a name inside curly braces to declare a url path parameter. For example <code>/user/{user_id}</code> defines a path where the <code>user_id</code> is a path parameter or a  variable which will be injected into the handler function (actually callable). Here’s a quick example:</p>
<pre>from apistar import Route
from apistar.frameworks.wsgi import WSGIApp as App
def user_profile(user_id: int):
 return {'message': 'Your profile id is: {}'.format(user_id)}
routes = [
 Route('/user/{user_id}', 'GET', user_profile),
]
app = App(routes=routes)
if __name__ == '__main__':
 app.main()</pre>
<p>If we visit <code>http://127.0.0.1:8080/user/23</code> we will get a response like this:</p>
<pre>{"message": "Your profile id is: 23"}</pre>
<p>But if we try to visit <code>http://127.0.0.1:8080/user/some_string</code> – it will not match. Because the <code>user_profile</code> function we defined, we added a type hint for the <code>user_id</code> parameter. If it’s not integer, the path doesn’t match. But if we go ahead and delete the type hint and just use <code>user_profile(user_id)</code>, it will match this url. This is again API Star is being smart and taking advantages of typing.</p>
<h5>Including / Grouping Routes</h5>
<p>Sometimes it might make sense to group certain urls together. Say we have a <code>user</code> module that deals with user related functionality. It might be better to group all the user related endpoints under the <code>/user</code> path. For example – <code>/user/new</code>, <code>/user/1</code>, <code>/user/1/update</code> and what not. We can easily create our handlers and routes in a separate module or package even and then include them in our own routes.</p>
<p>Let’s create a new module named <code>user</code>, the file name would be <code>user.py</code>. Let’s put these codes in this file:</p>
<pre>from apistar import Route
def user_new():
 return {"message": "Create a new user"}
def user_update(user_id: int):
 return {"message": "Update user #{}".format(user_id)}
def user_profile(user_id: int):
 return {"message": "User Profile for: {}".format(user_id)}
user_routes = [
 Route("/new", "GET", user_new),
 Route("/{user_id}/update", "GET", user_update),
 Route("/{user_id}/profile", "GET", user_profile),
]</pre>
<p>Now we can import our <code>user_routes</code> from within our main app file and use it like this:</p>
<pre>from apistar import Include
from apistar.frameworks.wsgi import WSGIApp as App
from user import user_routes
routes = [
 Include("/user", user_routes)
]
app = App(routes=routes)
if __name__ == '__main__':
 app.main()
</pre>
<p>Now <code>/user/new</code> will delegate to <code>user_new</code> function.</p>
<h3>Accessing Query String / Query Parameters</h3>
<p>Any parameters passed in the query parameters can be injected directly into handler function. Say for the url <code>/call?phone=1234</code>, the handler function can define a <code>phone</code> parameter and it will receive the value from the query string / query parameters. If the url query string doesn’t include a value for <code>phone</code>, it will get <code>None</code> instead. We can also set a default value to the parameter like this:</p>
<pre>def welcome(name=None):
 if name is None:
 return {'message': 'Welcome to API Star!'}
 return {'message': 'Welcome to API Star, %s!' % name}</pre>
<p>In the above example, we set a default value to <code>name</code> which is <code>None</code> anyway.</p>
<h3>Injecting Objects</h3>
<p>By type hinting a request handler, we can have different objects injected into our views. Injecting request related objects can be helpful for accessing them directly from inside the handler. There are several built in objects in the <code>http</code> package from API Star itself. We can also use it’s type system to create our own custom objects and have them injected into our functions. API Star also does data validation based on the constraints specified.</p>
<p>Let’s define our own <code>User</code> type and have it injected in our request handler:</p>
<pre>from apistar import Include, Route
from apistar.frameworks.wsgi import WSGIApp as App
from apistar import typesystem
class User(typesystem.Object):
 properties = {
 'name': typesystem.string(max_length=100),
 'email': typesystem.string(max_length=100),
 'age': typesystem.integer(maximum=100, minimum=18)
 }
 required = ["name", "age", "email"]
def new_user(user: User):
 return user
routes = [
 Route('/', 'POST', new_user),
]
app = App(routes=routes)
if __name__ == '__main__':
 app.main()
</pre>
<p>Now if we send this request:</p>
<pre>curl -X POST \
 http://127.0.0.1:8080/ \
 -H 'Cache-Control: no-cache' \
 -H 'Content-Type: application/json' \
 -d '{"name": "masnun", "email": "masnun@gmail.com", "age": 12}'</pre>
<p>Guess what happens? We get an error saying age must be equal to or greater than 18. The type system is allowing us intelligent data validation as well. If we enable the <code>docs</code> url, we will also get these parameters automatically documented there.</p>
<h3>Sending a Response</h3>
<p>If you have noticed so far, we can just pass a dictionary and it will be JSON encoded and returned by default. However, we can set the status code and any additional headers by using the <code>Response</code> class from <code>apistar</code>. Here’s a quick example:</p>
<pre>from apistar import Route, Response
from apistar.frameworks.wsgi import WSGIApp as App
def hello():
 return Response(
 content="Hello".encode("utf-8"),
 status=200,
 headers={"X-API-Framework": "API Star"},
 content_type="text/plain"
 )
routes = [
 Route('/', 'GET', hello),
]
app = App(routes=routes)
if __name__ == '__main__':
 app.main()</pre>
<p>It should send a plain text response along with a custom header. Please note that the <code>content</code> should be bytes, not string. That’s why I encoded it.</p>
<h3>Moving On</h3>
<p>I just walked through some of the features of API Star. There’s a lot more of cool stuff in API Star. I do recommend going through the <a href="https://github.com/encode/apistar">Github Readme</a> for learning more about different features offered by this excellent framework. I shall also try to cover short, focused tutorials on API Star in the coming days.	</p>
	</div>||||I||||Skip to the content
 Search

 Abu Ashraf Masnun

 Tales of a software craftsman
 Menu
 * Home
 * About Me
 * Tutorials
 * Codes
 * Contact
 Search
 Search for: Close search
 Close Menu
 * Home
 * About Me
 * Tutorials
 * Codes
 * Contact
 Categories
 General Python

 Auto incrementing IDs for MongoDB

 * Post author By masnun
 * Post date December 6, 2018

 If you’re familiar with relational databases like MySQL or PostgreSQL, you’re probably also familiar with auto incrementing IDs. You select a primary key for a table and make it auto incrementing. Every row you insert afterwards, each of them gets a new ID, automatically incremented from the last one. We don’t have to keep track of what number comes next or ensure the atomic nature of this operation (what happens if two different client wants to insert a new row at the very same time? do they both get the same id?). This can be very useful where sequential, numeric IDs are essential. For example, let’s say we’re building a url shortener. We can base62 encode the ID of the url id to quickly generate a short slug for that long url.

 Fast forward to MongoDB, the popular NoSQL database doesn’t have any equivalent to sequential IDs. It’s true that you can insert anything unique as the required _id field of a mongodb document, so you can take things to your hand and try to insert unique ids yourselves. But you have to ensure the uniqueness and atomicity of the operation.

 A very popular work around to this is to create a separate mongodb collection. Then maintain documents with a numeric value to keep track of your auto incrementing IDs. Now, every time we want to insert a new document that needs a unique ID, we come back to this collection, use the $inc operator to atomically increment this number and then use the incremented number as the unique id for our new document.

 Let me give an example, say we have an messages collection. Each new message needs a new, sequential ID. We create a new collection named sequences. Each document in this sequences collection will hold the last used ID for a collection. So, for tracking the unique ID in the messages collection, we create a new document in the sequences collection like this:

 {
 "_id" : "messages",
 "value" : 0
 }

 Next, we will write a function that can give us the next sequential ID for a collection by it’s name. The code is in Python, using PyMongo library.

 def get_sequence(name):
 collection = db.sequences
 document = collection.find_one_and_update({"_id": name}, {"$inc": {"value": 1}}, return_document=True)
 
 return document["value"]

 If we need the next auto incrementing ID for the messages collection, we can call it like this:

 {"_id": get_sequence("messages")}

 Find and Modify – Deprecated

 If you have searched on Google, you might have come across many StackOverflow answers as well as individual blog posts which refer to findAndModify() call (find_and_modify in Pymongo). This was the way to do things. But it’s deprecated now, so please use the new find_one_and_update function now.

 (How) Does this scale?

 We would only call the get_sequence function before inserting a new mongo document. The function uses the $inc operator which is atomic in nature. Mongo guarantees this. So even if 100s of different clients trying to increment the value for the same document, they will be all applied one after one. So each value they get will be unique, new IDs.

 I personally haven’t been able to test this strategy at a larger scale but according to people on StackOverflow and other forums, people have scaled this to thousands and millions of users. So I guess it’s pretty safe.

 * Tags mongodb
 Categories
 Python

 API Star: Python 3 API Framework

 * Post author By masnun
 * Post date February 3, 2018

 For building quick APIs in Python, I have mostly depended on Flask. Recently I came across a new API framework for Python 3 named “API Star” which seemed really interesting to me for several reasons. Firstly the framework embraces modern Python features like type hints and asyncio. And then it goes ahead and uses these features to provide awesome development experience for us, the developers. We will get into those features soon but before we begin, I would like to thank Tom Christie for all the work he has put into Django REST Framework and now API Star.

 Now back to API Star – I feel very productive in the framework. I can choose to write async codes based on asyncio or I can choose a traditional backend like WSGI. It comes with a command line tool – apistar to help us get things done faster. There’s (optional) support for both Django ORM and SQLAlchemy. There’s a brilliant type system that enables us to define constraints on our input and output and from these, API Star can auto generate api schemas (and docs), provide validation and serialization feature and a lot more. Although API Star is heavily focused on building APIs, you can also build web applications on top of it fairly easily. All these might not make proper sense until we build something all by ourselves.

 Getting Started

 We will start by installing API Star. It would be a good idea to create a virtual environment for this exercise. If you don’t know how to create a virtualenv, don’t worry and go ahead.

 pip install apistar

 If you’re not using a virtual environment or the pip command for your Python 3 is called pip3, then please use pip3 install apistar instead.

 Once we have the package installed, we should have access to the apistar command line tool. We can create a new project with it. Let’s create a new project in our current directory.

 apistar new .

 Now we should have two files created – app.py – which contains the main application and then test.py for our tests. Let’s examine our app.py file:

 from apistar import Include, Route
 from apistar.frameworks.wsgi import WSGIApp as App
 from apistar.handlers import docs_urls, static_urls
 
 
 def welcome(name=None):
 if name is None:
 return {'message': 'Welcome to API Star!'}
 return {'message': 'Welcome to API Star, %s!' % name}
 
 
 routes = [
 Route('/', 'GET', welcome),
 Include('/docs', docs_urls),
 Include('/static', static_urls)
 ]
 
 app = App(routes=routes)
 
 
 if __name__ == '__main__':
 app.main()

 Before we dive into the code, let’s run the app and see if it works. If we navigate to http://127.0.0.1:8080/ we will get this following response:

 {"message": "Welcome to API Star!"}

 And if we navigate to: http://127.0.0.1:8080/?name=masnun

 {"message": "Welcome to API Star, masnun!"}

 Similarly if we navigate to: http://127.0.0.1:8080/docs/, we will see auto generated docs for our API.

 Now let’s look at the code. We have a welcome function that takes a parameter named name which has a default value of None. API Star is a smart api framework. It will try to find the name key in the url path or query string and pass it to our function. It also generates the API docs based on it. Pretty nice, no?

 We then create a list of Route and Include instances and pass the list to the App instance. Route objects are used to define custom user routing. Include , as the name suggests, includes/embeds other routes under the path provided to it.

 Routing

 Routing is simple. When constructing the App instance, we need to pass a list as the routes argument. This list should comprise of Route or Include objects as we just saw above. For Routes, we pass a url path, http method name and the request handler callable (function or otherwise). For the Include instances, we pass a url path and a list of Routes instance.

 Path Parameters

 We can put a name inside curly braces to declare a url path parameter. For example /user/{user_id} defines a path where the user_id is a path parameter or a variable which will be injected into the handler function (actually callable). Here’s a quick example:

 from apistar import Route
 from apistar.frameworks.wsgi import WSGIApp as App
 
 
 def user_profile(user_id: int):
 return {'message': 'Your profile id is: {}'.format(user_id)}
 
 
 routes = [
 Route('/user/{user_id}', 'GET', user_profile),
 ]
 
 app = App(routes=routes)
 
 if __name__ == '__main__':
 app.main()

 If we visit http://127.0.0.1:8080/user/23 we will get a response like this:

 {"message": "Your profile id is: 23"}

 But if we try to visit http://127.0.0.1:8080/user/some_string – it will not match. Because the user_profile function we defined, we added a type hint for the user_id parameter. If it’s not integer, the path doesn’t match. But if we go ahead and delete the type hint and just use user_profile(user_id), it will match this url. This is again API Star is being smart and taking advantages of typing.

 Including / Grouping Routes

 Sometimes it might make sense to group certain urls together. Say we have a user module that deals with user related functionality. It might be better to group all the user related endpoints under the /user path. For example – /user/new, /user/1, /user/1/update and what not. We can easily create our handlers and routes in a separate module or package even and then include them in our own routes.

 Let’s create a new module named user, the file name would be user.py. Let’s put these codes in this file:

 from apistar import Route
 
 
 def user_new():
 return {"message": "Create a new user"}
 
 
 def user_update(user_id: int):
 return {"message": "Update user #{}".format(user_id)}
 
 
 def user_profile(user_id: int):
 return {"message": "User Profile for: {}".format(user_id)}
 
 
 user_routes = [
 Route("/new", "GET", user_new),
 Route("/{user_id}/update", "GET", user_update),
 Route("/{user_id}/profile", "GET", user_profile),
 ]

 Now we can import our user_routes from within our main app file and use it like this:

 from apistar import Include
 from apistar.frameworks.wsgi import WSGIApp as App
 
 from user import user_routes
 
 routes = [
 Include("/user", user_routes)
 ]
 
 app = App(routes=routes)
 
 if __name__ == '__main__':
 app.main()
 

 Now /user/new will delegate to user_new function.

 Accessing Query String / Query Parameters

 Any parameters passed in the query parameters can be injected directly into handler function. Say for the url /call?phone=1234, the handler function can define a phone parameter and it will receive the value from the query string / query parameters. If the url query string doesn’t include a value for phone, it will get None instead. We can also set a default value to the parameter like this:

 def welcome(name=None):
 if name is None:
 return {'message': 'Welcome to API Star!'}
 return {'message': 'Welcome to API Star, %s!' % name}

 In the above example, we set a default value to name which is None anyway.

 Injecting Objects

 By type hinting a request handler, we can have different objects injected into our views. Injecting request related objects can be helpful for accessing them directly from inside the handler. There are several built in objects in the http package from API Star itself. We can also use it’s type system to create our own custom objects and have them injected into our functions. API Star also does data validation based on the constraints specified.

 Let’s define our own User type and have it injected in our request handler:

 from apistar import Include, Route
 from apistar.frameworks.wsgi import WSGIApp as App
 from apistar import typesystem
 
 
 class User(typesystem.Object):
 properties = {
 'name': typesystem.string(max_length=100),
 'email': typesystem.string(max_length=100),
 'age': typesystem.integer(maximum=100, minimum=18)
 }
 
 required = ["name", "age", "email"]
 
 
 def new_user(user: User):
 return user
 
 
 routes = [
 Route('/', 'POST', new_user),
 ]
 
 app = App(routes=routes)
 
 if __name__ == '__main__':
 app.main()
 

 Now if we send this request:

 curl -X POST \
 http://127.0.0.1:8080/ \
 -H 'Cache-Control: no-cache' \
 -H 'Content-Type: application/json' \
 -d '{"name": "masnun", "email": "masnun@gmail.com", "age": 12}'

 Guess what happens? We get an error saying age must be equal to or greater than 18. The type system is allowing us intelligent data validation as well. If we enable the docs url, we will also get these parameters automatically documented there.

 Sending a Response

 If you have noticed so far, we can just pass a dictionary and it will be JSON encoded and returned by default. However, we can set the status code and any additional headers by using the Response class from apistar. Here’s a quick example:

 from apistar import Route, Response
 from apistar.frameworks.wsgi import WSGIApp as App
 
 
 def hello():
 return Response(
 content="Hello".encode("utf-8"),
 status=200,
 headers={"X-API-Framework": "API Star"},
 content_type="text/plain"
 )
 
 
 routes = [
 Route('/', 'GET', hello),
 ]
 
 app = App(routes=routes)
 
 if __name__ == '__main__':
 app.main()

 It should send a plain text response along with a custom header. Please note that the content should be bytes, not string. That’s why I encoded it.

 Moving On

 I just walked through some of the features of API Star. There’s a lot more of cool stuff in API Star. I do recommend going through the Github Readme for learning more about different features offered by this excellent framework. I shall also try to cover short, focused tutorials on API Star in the coming days.

 * Tags REST API
 Categories
 Python

 Getting Started with Pipenv

 * Post author By masnun
 * Post date November 26, 2017

 If you’re a Python developer, you probably know about pip and the different environment management solutions like virtualenv or venv. The pip tool is currently the standard way to install a Python package. Virtualenv has been a popular way of isolating Python environments for a long time. Pipenv combines the very best of these tools and brings us the one true way to install packages while keeping the dependencies from each project isolated. It claims to have brought the very best from all other packaging worlds (the package manager for other languages / runtimes / frameworks) to the Python world. From what I have seen so far, that claim is quite valid. And it does support Windows pretty well too.

 How does Pipenv work?

 Pipenv works by creating and managing a virtualenv and a Pipfile for your project. When we install / remove Python packages, the changes are reflected in the Pipfile. It also generates a lock file named Pipfile.lock which is used to lock the version of dependencies and help us produce deterministic builds when needed. In a typical virtualenv setup, we usually create and manage the virtualenv ourselves. Then we activate the environment and pip just installs / uninstalls from that particular virtual environment. Packages like virtualenvwrapper helps us easily create and activate virtualenv with some of it’s handy features. But pipenv takes things further by automating a large part of that. The Pipfile would also make more sense if you have used other packaging systems like Composer, npm, bundler etc.

 Getting Started

 We need to start by installing pipenv globally. We can install it using pip from PyPi:

 pip install pipenv

 Now let’s switch to our project directory and try installing a package:

 pipenv install flask

 When you first run the pipenv install command, you will notice it creates a virtualenv, Pipfile and Pipfile.lock for you. Feel free to go ahead inspect their contents. If you’re using an IDE like PyCharm and want to configure your project interpreter, it would be a good idea to note down the virtualenv path.

 Since we have installed Flask, let’s try and running a sample app. Here’s my super simple REST API built with Flask:

 from flask import Flask, jsonify
 
 app = Flask(__name__)
 
 
 @app.route('/')
 def hello_world():
 return jsonify({"message": "Hello World!"})

 Assuming that you have the FLASK_APP environment variable set to app.py (which contains the above code), we can just run the app like this:

 pipenv run flask run

 Any executables in the current environment can be run using the pipenv run command. But I know what you might be thinking – we want to do just flask run, not use the entire, long command. That’s easy too. We just need to activate the virtualenv with this command:

 pipenv shell

 Now you can just do flask run or in fact run any executables in the way we’re used to doing.

 Handling Dependencies

 We can install and uninstall packages using the install and uninstall commands. When we install a new package, it’s added to our Pipfile and the lock file is updated as well. When we uninstall a package, the Pipfile and the lock files are again updated to reflect the change. The update command uninstalls the packages and installs them again so we have the latest updates.

 If you would like to check your dependency graph, just use the graph command which will print out the dependencies in a nice format, kind of like this:

 PS C:\Users\Masnun\Documents\Python\pipenvtest> pipenv graph
 celery==4.1.0
 - billiard [required: >=3.5.0.2,<3.6.0, installed: 3.5.0.3]
 - kombu [required: >=4.0.2,<5.0, installed: 4.1.0]
 - amqp [required: >=2.1.4,<3.0, installed: 2.2.2]
 - vine [required: >=1.1.3, installed: 1.1.4]
 - pytz [required: >dev, installed: 2017.3]
 Flask==0.12.2
 - click [required: >=2.0, installed: 6.7]
 - itsdangerous [required: >=0.21, installed: 0.24]
 - Jinja2 [required: >=2.4, installed: 2.10]
 - MarkupSafe [required: >=0.23, installed: 1.0]
 - Werkzeug [required: >=0.7, installed: 0.12.2]

 Pipenv is Awesome!

 Trust me, it is! It packs a lot of cool and useful features that can help gear up your Python development workflow. There are just too many to cover in a blog post. I would recommend checking out the Pipenv Docs to get familiar with it more.

 Posts navigation

 ← Newer Posts 1 2 … 124 Older Posts →
 Search
 Search

 Recent Posts

 * Auto incrementing IDs for MongoDB
 * API Star: Python 3 API Framework
 * Getting Started with Pipenv
 * Deploying A Flask based REST API to AWS Lambda (Serverless) using Zappa
 * Golang: Interface

 Recent Comments

 1. Umut Çağdaş Coşkun on Django REST Framework: Remember to disable Web Browsable API in Production
 2. kelvin on Using annotations in PHP with Doctrine Annotation Reader
 3. Junho on Python: A quick introduction to the concurrent.futures module
 4. Bushra Mustofa on Python/Django: Running multiple commands in subprocesses
 5. Flask-Httpauth Vs Flask-Login - LoginCrunch on Securing REST APIs: Basic HTTP Authentication with Python / Flask

 Archives

 * December 2018
 * February 2018
 * November 2017
 * October 2017
 * August 2017
 * July 2017
 * June 2017
 * May 2017
 * February 2017
 * September 2016
 * June 2016
 * May 2016
 * April 2016
 * March 2016
 * February 2016
 * January 2016
 * December 2015
 * November 2015
 * October 2015
 * September 2015
 * July 2015
 * June 2015
 * May 2015
 * February 2015
 * December 2014
 * September 2014
 * August 2014
 * July 2014
 * June 2014
 * May 2014
 * April 2014
 * March 2014
 * February 2014
 * January 2014
 * December 2013
 * September 2013
 * August 2013
 * July 2013
 * June 2013
 * May 2013
 * February 2013
 * November 2012
 * October 2012
 * August 2012
 * July 2012
 * June 2012
 * May 2012
 * April 2012
 * March 2012
 * February 2012
 * January 2012
 * December 2011
 * November 2011
 * October 2011
 * September 2011
 * August 2011
 * July 2011
 * June 2011
 * May 2011
 * April 2011
 * March 2011
 * February 2011
 * January 2011
 * December 2010
 * November 2010
 * October 2010
 * September 2010
 * July 2010
 * June 2010
 * May 2010
 * April 2010
 * March 2010
 * February 2010
 * January 2010
 * December 2009
 * November 2009
 * October 2009
 * September 2009
 * August 2009
 * June 2009
 * May 2009
 * April 2009
 * March 2009
 * February 2009
 * January 2007
 * December 2006
 * June 2006

 Categories

 * Bangla
 * Clojure
 * Data Science
 * Django
 * Javascript
 * Linux
 * Mac
 * MySQL
 * NodeJS
 * Personal
 * PHP
 * Python
 * Screencast
 * Series
 * Uncategorized
 * Work

 © 2023 Abu Ashraf Masnun

 Powered by WordPress

 To the top ↑ Up ↑
For building quick APIs in Python, I have mostly depended on Flask. Recently I came across a new API framework for Python 3 named “API Star” which seemed really interesting to me for several reasons. Firstly the framework embraces modern Python features like type hints and asyncio. And then it goes ahead and uses these features to provide awesome development experience for us, the developers. We will get into those features soon but before we begin, I would like to thank Tom Christie for all the work he has put into Django REST Framework and now API Star.
Now back to API Star – I feel very productive in the framework. I can choose to write async codes based on asyncio or I can choose a traditional backend like WSGI. It comes with a command line tool – apistar to help us get things done faster. There’s (optional) support for both Django ORM and SQLAlchemy.  There’s a brilliant type system that enables us to define constraints on our input and output and from these, API Star can auto generate api schemas (and docs), provide validation and serialization feature and a lot more. Although API Star is heavily focused on building APIs, you can also build web applications on top of it fairly easily. All these might not make proper sense until we build something all by ourselves.
Getting Started
We will start by installing API Star. It would be a good idea to create a virtual environment for this exercise. If you don’t know how to create a virtualenv, don’t worry and go ahead.
pip install apistar
If you’re not using a virtual environment or the pip command for your Python 3 is called pip3, then please use pip3 install apistar instead.
Once we have the package installed, we should have access to the apistar command line tool. We can create a new project with it. Let’s create a new project in our current directory.
apistar new .
Now we should have two files created – app.py – which contains the main application and then test.py for our tests. Let’s examine our app.py file:
from apistar import Include, Route
from apistar.frameworks.wsgi import WSGIApp as App
from apistar.handlers import docs_urls, static_urls
def welcome(name=None):
 if name is None:
 return {'message': 'Welcome to API Star!'}
 return {'message': 'Welcome to API Star, %s!' % name}
routes = [
 Route('/', 'GET', welcome),
 Include('/docs', docs_urls),
 Include('/static', static_urls)
]
app = App(routes=routes)
if __name__ == '__main__':
 app.main()
Before we dive into the code, let’s run the app and see if it works. If we navigate to http://127.0.0.1:8080/ we will get this following response:
{"message": "Welcome to API Star!"}
And if we navigate to: http://127.0.0.1:8080/?name=masnun 
{"message": "Welcome to API Star, masnun!"}
Similarly if we navigate to: http://127.0.0.1:8080/docs/, we will see auto generated docs for our API.
Now let’s look at the code. We have a welcome function that takes a parameter named name which has a default value of None. API Star is a smart api framework. It will try to find the name key in the url path or query string and pass it to our function. It also generates the API docs based on it. Pretty nice, no?
We then create a list of Route and Include instances and pass the list to the App instance. Route objects are used to define custom user routing. Include , as the name suggests, includes/embeds other routes under the path provided to it.
Routing
Routing is simple. When constructing the App instance, we need to pass a list as the routes argument. This list should comprise of Route or Include objects as we just saw above. For Routes, we pass a url path, http method name and the request handler callable (function or otherwise). For the Include instances, we pass a url path and a list of Routes instance.
Path Parameters
We can put a name inside curly braces to declare a url path parameter. For example /user/{user_id} defines a path where the user_id is a path parameter or a  variable which will be injected into the handler function (actually callable). Here’s a quick example:
from apistar import Route
from apistar.frameworks.wsgi import WSGIApp as App
def user_profile(user_id: int):
 return {'message': 'Your profile id is: {}'.format(user_id)}
routes = [
 Route('/user/{user_id}', 'GET', user_profile),
]
app = App(routes=routes)
if __name__ == '__main__':
 app.main()
If we visit http://127.0.0.1:8080/user/23 we will get a response like this:
{"message": "Your profile id is: 23"}
But if we try to visit http://127.0.0.1:8080/user/some_string – it will not match. Because the user_profile function we defined, we added a type hint for the user_id parameter. If it’s not integer, the path doesn’t match. But if we go ahead and delete the type hint and just use user_profile(user_id), it will match this url. This is again API Star is being smart and taking advantages of typing.
Including / Grouping Routes
Sometimes it might make sense to group certain urls together. Say we have a user module that deals with user related functionality. It might be better to group all the user related endpoints under the /user path. For example – /user/new, /user/1, /user/1/update and what not. We can easily create our handlers and routes in a separate module or package even and then include them in our own routes.
Let’s create a new module named user, the file name would be user.py. Let’s put these codes in this file:
from apistar import Route
def user_new():
 return {"message": "Create a new user"}
def user_update(user_id: int):
 return {"message": "Update user #{}".format(user_id)}
def user_profile(user_id: int):
 return {"message": "User Profile for: {}".format(user_id)}
user_routes = [
 Route("/new", "GET", user_new),
 Route("/{user_id}/update", "GET", user_update),
 Route("/{user_id}/profile", "GET", user_profile),
]
Now we can import our user_routes from within our main app file and use it like this:
from apistar import Include
from apistar.frameworks.wsgi import WSGIApp as App
from user import user_routes
routes = [
 Include("/user", user_routes)
]
app = App(routes=routes)
if __name__ == '__main__':
 app.main()
Now /user/new will delegate to user_new function.
Accessing Query String / Query Parameters
Any parameters passed in the query parameters can be injected directly into handler function. Say for the url /call?phone=1234, the handler function can define a phone parameter and it will receive the value from the query string / query parameters. If the url query string doesn’t include a value for phone, it will get None instead. We can also set a default value to the parameter like this:
def welcome(name=None):
 if name is None:
 return {'message': 'Welcome to API Star!'}
 return {'message': 'Welcome to API Star, %s!' % name}
In the above example, we set a default value to name which is None anyway.
Injecting Objects
By type hinting a request handler, we can have different objects injected into our views. Injecting request related objects can be helpful for accessing them directly from inside the handler. There are several built in objects in the http package from API Star itself. We can also use it’s type system to create our own custom objects and have them injected into our functions. API Star also does data validation based on the constraints specified.
Let’s define our own User type and have it injected in our request handler:
from apistar import Include, Route
from apistar.frameworks.wsgi import WSGIApp as App
from apistar import typesystem
class User(typesystem.Object):
 properties = {
 'name': typesystem.string(max_length=100),
 'email': typesystem.string(max_length=100),
 'age': typesystem.integer(maximum=100, minimum=18)
 }
 required = ["name", "age", "email"]
def new_user(user: User):
 return user
routes = [
 Route('/', 'POST', new_user),
]
app = App(routes=routes)
if __name__ == '__main__':
 app.main()
Now if we send this request:
curl -X POST \
 http://127.0.0.1:8080/ \
 -H 'Cache-Control: no-cache' \
 -H 'Content-Type: application/json' \
 -d '{"name": "masnun", "email": "masnun@gmail.com", "age": 12}'
Guess what happens? We get an error saying age must be equal to or greater than 18. The type system is allowing us intelligent data validation as well. If we enable the docs url, we will also get these parameters automatically documented there.
Sending a Response
If you have noticed so far, we can just pass a dictionary and it will be JSON encoded and returned by default. However, we can set the status code and any additional headers by using the Response class from apistar. Here’s a quick example:
from apistar import Route, Response
from apistar.frameworks.wsgi import WSGIApp as App
def hello():
 return Response(
 content="Hello".encode("utf-8"),
 status=200,
 headers={"X-API-Framework": "API Star"},
 content_type="text/plain"
 )
routes = [
 Route('/', 'GET', hello),
]
app = App(routes=routes)
if __name__ == '__main__':
 app.main()
It should send a plain text response along with a custom header. Please note that the content should be bytes, not string. That’s why I encoded it.
Moving On
I just walked through some of the features of API Star. There’s a lot more of cool stuff in API Star. I do recommend going through the Github Readme for learning more about different features offered by this excellent framework. I shall also try to cover short, focused tutorials on API Star in the coming days.
Abu Ashraf Masnun
http://masnun.com/
https://adambard.com/blog/
Adam Bard
2020-07-26
None
<div><div><article>
<p>July 26, 2020</p>
<h2>
<a href="/blog/transitioning-from-docker-machines-to-kubernetes/">Transitioning Later for Reddit from Docker Machines to Digital Ocean Kubernetes</a>
</h2>
<p>I recently completed a migration from my old docker-machine-based infrastructure (so 2016!) to <a href="https://www.digitalocean.com/products/kubernetes/?refcode=2c5db3effb6f&amp;utm_campaign=Referral_Invite&amp;utm_medium=Referral_Program&amp;utm_source=CopyPaste">Digital Oceanâs hosted kubernetes service</a>.
The bulk of the work was completed in a single evening. It went almost seamlessly, but not so seamlessly that I wasnât able to learn a few things.
The goal of this writeup is not to serve as comprehensive documentation, but hopefully youâll take away some ideas about how to get your
own setup rolling.</p>
<p>
<a href="/blog/transitioning-from-docker-machines-to-kubernetes/">Read more »</a>
</p>
</article>
<article>
<p>December 17, 2018</p>
<h2>
<a href="/blog/getting-a-handle-on-reduce/">Getting a handle on Reduce</a>
</h2>
<p>Although traditional algol-inspired languages remain dominant, over the past
decade or so, functional programming techniques utilizing
immutable data structures have seen widespread adoption in a variety of
domains. From React and friends on the frontend to Map-Reduce and the
vast ecosystem of thus-inspired data warehousing systems that power some of
the worldâs largest software applications, the ability to navigate
such applications is now a must-have skill for many programmers.</p>
<p>The subject of todayâs article is a <a href="https://en.wikipedia.org/wiki/Higher-order_function">higher-order function</a> called <code>reduce</code>,
also known as <code>fold</code>, <code>aggregate</code>, and others. Reduce is a fundamentally
useful tool, but I find that many developers have trouble grasping and
using it. My hope here is to make an approachable introduction to <code>reduce</code>.</p>
<p>
<a href="/blog/getting-a-handle-on-reduce/">Read more »</a>
</p>
</article>
<article>
<p>May 23, 2018</p>
<h2>
<a href="/blog/programatically-liquidating-my-steam-inventory/">Programatically Liquidating my Steam Inventory</a>
</h2>
<p>I donât know about you, but every so often my Steam Inventory gets a bit out of control.
I have no use for Donât Starve trading cards, but itâs not really worth my time to individually
sell each item for a few cents. However, what is worth my time, apparently, is creating a
script to do it for me. This turned out to be quite a winding road, which Iâve documented here.</p>
<p>
<a href="/blog/programatically-liquidating-my-steam-inventory/">Read more »</a>
</p>
</article>
<article>
<p>February 18, 2018</p>
<h2>
<a href="/blog/stream-processing-core-async/">One-box stream processing with CSP</a>
</h2>
<p>If youâre like me (that is, employed by an ad tech company), stream processing is usually
associated with frameworks like <a href="https://storm.apache.org/">Storm</a>,
<a href="https://flink.apache.org/">Flink</a>, <a href="https://spark.apache.org/streaming/">Spark Streaming</a>, and other such solutions. However, a
lot of real-life software can be described as stream processing â data comes
in one end, is transformed or aggregated, and goes somewhere else. Many of these
workloads donât justify the overhead of a stream processor, but that doesnât mean
they canât benefit from some of the lessons of stream processing systems.</p>
<p>
<a href="/blog/stream-processing-core-async/">Read more »</a>
</p>
</article>
<article>
<p>December 19, 2017</p>
<h2>
<a href="/blog/later-for-reddit-in-review/">Running a Side Business as a Programmer: The story of </a><a href="https://laterforreddit.com/">Later for Reddit</a>
</h2>
<p>For the past 5 years, Iâve run a website called
<a href="https://laterforreddit.com/">Later for Reddit</a>. For the past 2 years, itâs actually
been making money. Hereâs how that happened.</p>
<p>I used to spend a lot of time on Reddit (insert Mitch Hedberg joke here).
I also used to spend a lot...</p>
<p>
<a href="/blog/later-for-reddit-in-review/">Read more »</a>
</p>
</article>
<article>
<p>September 16, 2017</p>
<h2>
<a href="/blog/javascript-design-pattern-binding-functions/">Javascript FP design pattern: Binding functions</a>
</h2>
<p>Ok, ok, monads have been done. Theyâve <a href="/blog/acceptable-error-handling-in-clojure/">even</a>
been <a href="/blog/introducing-failjure/">done</a> by <a href="/blog/oo-vs-functional-form-validation/">me before</a>.
But, functional programming is still popular, monads are still very useful for programming
in an FP style, and people are entering the field every day. So hereâs another attempt at
an approachable introduction to <em>usefully</em> apply this pattern to everyday coding problems.</p>
<p>
<a href="/blog/javascript-design-pattern-binding-functions/">Read more »</a>
</p>
</article>
<article>
<p>September 3, 2017</p>
<h2>
<a href="/blog/reason-vs-elm-vs-typescript/">Bucklescript vs Elm vs Typescript: Typed Javascript showdown!</a>
</h2>
<p>You, a web developer, have probably heard of <a href="https://www.typescriptlang.org/index.html">Typescript</a>, may have heard of <a href="http://elm-lang.org/">Elm</a>, and
you might even have heard of <a href="https://bucklescript.github.io/bucklescript/Manual.html">Bucklescript</a>/<a href="reasonml.github.io">ReasonML</a> as well. Each of these
represents a compiles-to-javascript language with strong type support, but each has some different opinions,
philosophies, and features.</p>
<p>In this article Iâll walk you through the features of each of these options and compare their build ecosystems,
editor tooling, and javascript interop. In addition, as is my habit, Iâve written up a small example in each of these languages. (Well, actually,
Iâve taken one of Elmâs small demos and re-created it in the other two). Iâll discuss my impressions
of the tooling and the coding experience along the way.</p>
<p>
<a href="/blog/reason-vs-elm-vs-typescript/">Read more »</a>
</p>
</article>
<article>
<p>August 28, 2017</p>
<h2>
<a href="/blog/getting-started-with-ocaml/">OCaml for the impatient</a>
</h2>
<p>Ocaml is a rad language with some regrettably underdocumented parts. Or, perhaps, overdocumented â
since OCaml attracts people who are interested in rigorous correctness, what documentation exists tends to be rigorous
and correct, and also tedious. This guide will be none of those. Letâs go!</p>
<p>
<a href="/blog/getting-started-with-ocaml/">Read more »</a>
</p>
</article>
<article>
<p>August 13, 2017</p>
<h2>
<a href="/blog/on-library-versioning/">Best Practices for Javascript Library Versioning</a>
</h2>
<p>I spend most of my time on this blog talking about Clojure, but IRL I work on
a lot of projects written in a lot of languages. Lately, much of my time has been spent writing and maintaining
React-based frontends in Javascript.</p>
<p>As far as I can tell, no language has solved the problem of dependency management. Some
libraries are better than others, but upgrading dependencies is always an uncomfortable
experience. That being said, Iâve <em>never</em> experienced more pain doing so than in Javascriptâs ecosystem.</p>
<p>
<a href="/blog/on-library-versioning/">Read more »</a>
</p>
</article>
<article>
<p>August 6, 2017</p>
<h2>
<a href="/blog/domain-modeling-with-clojure-spec/">Domain modelling with clojure.spec</a>
</h2>
<p><a href="https://clojure.org/about/spec">Clojure.spec</a> is, among other things, Clojureâs official answer to tools like <a href="https://github.com/clojure/core.typed">Typed Clojure</a> and
<a href="https://github.com/plumatic/schema">Plumaticâs Schema</a>. It represents an attempt to apply some validation to
data and functions, without compromising Clojureâs dynamism and data-is-data philosophy.
In this post, Iâll be working through a sample program by first outlining and modelling
it with the help of clojure.spec, then using spec to guide me while I develop the implementation.</p>
<p>
<a href="/blog/domain-modeling-with-clojure-spec/">Read more »</a>
</p>
</article>
<article>
<p>March 23, 2017</p>
<h2>
<a href="/blog/what-programming-language/">What programming language should I learn? (2017 edition)</a>
</h2>
<p>I occasionally find someone asking this question in my inbox. Iâve learned
that for every person that bothers there are a bunch more with the same question,
so I thought Iâd take a break from relentlessly plugging my side hustle
<a href="https://laterforreddit.com/">Later for Reddit</a> to answer it publicly.</p>
<p>
<a href="/blog/what-programming-language/">Read more »</a>
</p>
</article>
<article>
<p>November 4, 2016</p>
<h2>
<a href="/blog/ideas/">On Advice (featuring just one piece of it)</a>
</h2>
<p>Iâm perfectly comfortable making wild assertions about which
programming technique is better than which other one, but
when it comes to handing out general life or career advice, Iâm a lot stingier
than your average blogger <sup><a href="#footnote-1">1</a></sup>. Even if I was wildly...</p>
<p>
<a href="/blog/ideas/">Read more »</a>
</p>
</article>
<article>
<p>August 23, 2016</p>
<h2>
<a href="/blog/basic-website-monitoring-with-riemann/">Basic site monitoring with Riemann</a>
</h2>
<p><a href="http://riemann.io/">Riemann</a> is a general-purpose event processing system, but its most typical application
is as a place to send and generate metrics about applications. I recently set up a Riemann
server for my personal projects, and I feel like my devops game is stepped up by 1000%.</p>
<p>Or, at very least, I feel like I know know about it as soon as one of my sites goes down.</p>
<p>
<a href="/blog/basic-website-monitoring-with-riemann/">Read more »</a>
</p>
</article>
<article>
<p>April 15, 2016</p>
<h2>
<a href="/blog/whats-in-a-namespace/">What’s in a namespace?</a>
</h2>
<p>Every well-behaved clojure source file starts with a namespace declaration.
The <code>ns</code> macro, as we all know, is responsible for declaring the namespace
to which the definitions in the rest of the file belong, and generally also
includes some requirements and imports and whatnot. But today (at
Clojure/West, shoutout!) Stuart Sierra made a passing reference to
the internals of <code>ns</code> during his talk that got me interested.</p>
<p>But what is a namespace <em>really</em>?</p>
<p>
<a href="/blog/whats-in-a-namespace/">Read more »</a>
</p>
</article>
<article>
<p>March 29, 2016</p>
<h2>
<a href="/blog/self-hosted-readability-on-heroku-with-nodejs/">Write and deploy your own self-hosted Readability clone</a>
</h2>
<p><a href="https://readability.org/">Readability.org</a> is a handy tool, but it also suffers from being useful, popular and free â
the service is (understandably) often down, or slow.</p>
<p>If youâre a heavy Readability user, it might interest you to know that there exist many
software libraries...</p>
<p>
<a href="/blog/self-hosted-readability-on-heroku-with-nodejs/">Read more »</a>
</p>
</article>
<article>
<p>March 27, 2016</p>
<h2>
<a href="/blog/composible-architecture-a-real-world-example/">Composible Architecture: A real-world example.</a>
</h2>
<p>I am the primary maintainer of a significant REST API written using <a href="http://flask-restful-cn.readthedocs.org/en/latest/">Flask-Restful</a>.
Flask-Restful is a pretty full-featured library for creating REST frameworks with Flask.
It features everything you need to manage routing to class-based resources, a library
for marshalling objects to JSON format, and reqparse, an argparse-inspired library
for writing input parsers.</p>
<p>
<a href="/blog/composible-architecture-a-real-world-example/">Read more »</a>
</p>
</article>
<article>
<p>March 26, 2016</p>
<h2>
<a href="/blog/react-in-the-small/">React.js in the small</a>
</h2>
<p>Iâm a big fan of accessible technologies. I get suspicious whenever I encounter
a framework or library that is ânot worth it for a small project,â because that
is a coded statement that means either âIt has a broad scope of features that
a small project wonât take advantage of,â or âit trades off simplicity or
user experience for performance or other reasons.â Accessibility also helps
open-source projects move from obscure to ubiquitous; if jQuery had required a
command-line tool to compile it when it came out, I can practically guarantee you wouldnât
have ever heard about it.</p>
<p>
<a href="/blog/react-in-the-small/">Read more »</a>
</p>
</article>
<article>
<p>March 19, 2016</p>
<h2>
<a href="/blog/the-web-is-a-mature-platform/">The Web is (finally) a mature platform</a>
</h2>
<p>I started my career as a âweb developerâ circa 2007. Iâd dabbled before
with Javascript while making sites in HTML/CSS as early as middle school,
but that was my first real job as a real software developer. Back
in those days, LAMP (Linux/Apache/MySQL/PHP) was about the only sensible way to
build a web application. You could use Java servlets, of course,
or build applications using CGI, but once PHP was stable and available
and straightforward, and it quickly became the obvious choice for a huge
range of applications.</p>
<p>
<a href="/blog/the-web-is-a-mature-platform/">Read more »</a>
</p>
</article>
<article>
<p>February 28, 2016</p>
<h2>
<a href="/blog/clojure-migration-libraries/">Clojure SQL Migration Libaries</a>
</h2>
<p>It is generally recommended, if youâre developing an application
that uses a database, that you manage changes to the schema of said
database using some sort of migration system or another. Luckily,
several intrepid developers in the clojure community have stepped
up with their own solutions to this problem â all of which end up
being very similar indeed. Read on for an overview of your options
and to find out which one is right for you.</p>
<p>
<a href="/blog/clojure-migration-libraries/">Read more »</a>
</p>
</article>
<article>
<p>February 20, 2016</p>
<h2>
<a href="/blog/plotting-reddit-post-frequencies-with-d3/">Plotting Reddit post frequencies with d3</a>
</h2>
<p>This week I added a better graph to <a href="https://www.redditlater.com/">Later (for Reddit)</a>âs
<a href="https://www.redditlater.com/analysis/#pics">post timing analysis page</a>. Previously,
the most common time and the most common day were plotted on separate bar charts. 
However, I noticed that I got called out on this in <a href="https://medium.com/@dleybzon/the-best-time-to-post-on-reddit-190762d4fc71">this article on Medium</a>,
which offered a pretty sensible alternative that I wasted no time in ripping off
wholesale. I still
have the bar graphs in, but I also added a 7x24 color-coded grid plot, allowing for a more
accurate view of exactly when the most popular posts were posted.</p>
<p>
<a href="/blog/plotting-reddit-post-frequencies-with-d3/">Read more »</a>
</p>
</article>
</div>||||I||||Adam Bard

Makes Websites

 July 26, 2020

 Transitioning Later for Reddit from Docker Machines to Digital Ocean Kubernetes

 I recently completed a migration from my old docker-machine-based infrastructure (so 2016!) to Digital Oceanâs hosted kubernetes service. The bulk of the work was completed in a single evening. It went almost seamlessly, but not so seamlessly that I wasnât able to learn a few things. The goal of this writeup is not to serve as comprehensive documentation, but hopefully youâll take away some ideas about how to get your own setup rolling.

 Read more »

 December 17, 2018

 Getting a handle on Reduce

 Although traditional algol-inspired languages remain dominant, over the past decade or so, functional programming techniques utilizing immutable data structures have seen widespread adoption in a variety of domains. From React and friends on the frontend to Map-Reduce and the vast ecosystem of thus-inspired data warehousing systems that power some of the worldâs largest software applications, the ability to navigate such applications is now a must-have skill for many programmers.

 The subject of todayâs article is a higher-order function called reduce, also known as fold, aggregate, and others. Reduce is a fundamentally useful tool, but I find that many developers have trouble grasping and using it. My hope here is to make an approachable introduction to reduce.

 Read more »

 May 23, 2018

 Programatically Liquidating my Steam Inventory

 I donât know about you, but every so often my Steam Inventory gets a bit out of control. I have no use for Donât Starve trading cards, but itâs not really worth my time to individually sell each item for a few cents. However, what is worth my time, apparently, is creating a script to do it for me. This turned out to be quite a winding road, which Iâve documented here.

 Read more »

 February 18, 2018

 One-box stream processing with CSP

 If youâre like me (that is, employed by an ad tech company), stream processing is usually associated with frameworks like Storm, Flink, Spark Streaming, and other such solutions. However, a lot of real-life software can be described as stream processing â data comes in one end, is transformed or aggregated, and goes somewhere else. Many of these workloads donât justify the overhead of a stream processor, but that doesnât mean they canât benefit from some of the lessons of stream processing systems.

 Read more »

 December 19, 2017

 Running a Side Business as a Programmer: The story of Later for Reddit

 For the past 5 years, Iâve run a website called Later for Reddit. For the past 2 years, itâs actually been making money. Hereâs how that happened.

 I used to spend a lot of time on Reddit (insert Mitch Hedberg joke here). I also used to spend a lot...

 Read more »

 September 16, 2017

 Javascript FP design pattern: Binding functions

 Ok, ok, monads have been done. Theyâve even been done by me before. But, functional programming is still popular, monads are still very useful for programming in an FP style, and people are entering the field every day. So hereâs another attempt at an approachable introduction to usefully apply this pattern to everyday coding problems.

 Read more »

 September 3, 2017

 Bucklescript vs Elm vs Typescript: Typed Javascript showdown!

 You, a web developer, have probably heard of Typescript, may have heard of Elm, and you might even have heard of Bucklescript/ReasonML as well. Each of these represents a compiles-to-javascript language with strong type support, but each has some different opinions, philosophies, and features.

 In this article Iâll walk you through the features of each of these options and compare their build ecosystems, editor tooling, and javascript interop. In addition, as is my habit, Iâve written up a small example in each of these languages. (Well, actually, Iâve taken one of Elmâs small demos and re-created it in the other two). Iâll discuss my impressions of the tooling and the coding experience along the way.

 Read more »

 August 28, 2017

 OCaml for the impatient

 Ocaml is a rad language with some regrettably underdocumented parts. Or, perhaps, overdocumented â since OCaml attracts people who are interested in rigorous correctness, what documentation exists tends to be rigorous and correct, and also tedious. This guide will be none of those. Letâs go!

 Read more »

 August 13, 2017

 Best Practices for Javascript Library Versioning

 I spend most of my time on this blog talking about Clojure, but IRL I work on a lot of projects written in a lot of languages. Lately, much of my time has been spent writing and maintaining React-based frontends in Javascript.

 As far as I can tell, no language has solved the problem of dependency management. Some libraries are better than others, but upgrading dependencies is always an uncomfortable experience. That being said, Iâve never experienced more pain doing so than in Javascriptâs ecosystem.

 Read more »

 August 6, 2017

 Domain modelling with clojure.spec

 Clojure.spec is, among other things, Clojureâs official answer to tools like Typed Clojure and Plumaticâs Schema. It represents an attempt to apply some validation to data and functions, without compromising Clojureâs dynamism and data-is-data philosophy. In this post, Iâll be working through a sample program by first outlining and modelling it with the help of clojure.spec, then using spec to guide me while I develop the implementation.

 Read more »

 March 23, 2017

 What programming language should I learn? (2017 edition)

 I occasionally find someone asking this question in my inbox. Iâve learned that for every person that bothers there are a bunch more with the same question, so I thought Iâd take a break from relentlessly plugging my side hustle Later for Reddit to answer it publicly.

 Read more »

 November 4, 2016

 On Advice (featuring just one piece of it)

 Iâm perfectly comfortable making wild assertions about which programming technique is better than which other one, but when it comes to handing out general life or career advice, Iâm a lot stingier than your average blogger 1. Even if I was wildly...

 Read more »

 August 23, 2016

 Basic site monitoring with Riemann

 Riemann is a general-purpose event processing system, but its most typical application is as a place to send and generate metrics about applications. I recently set up a Riemann server for my personal projects, and I feel like my devops game is stepped up by 1000%.

 Or, at very least, I feel like I know know about it as soon as one of my sites goes down.

 Read more »

 April 15, 2016

 What’s in a namespace?

 Every well-behaved clojure source file starts with a namespace declaration. The ns macro, as we all know, is responsible for declaring the namespace to which the definitions in the rest of the file belong, and generally also includes some requirements and imports and whatnot. But today (at Clojure/West, shoutout!) Stuart Sierra made a passing reference to the internals of ns during his talk that got me interested.

 But what is a namespace really?

 Read more »

 March 29, 2016

 Write and deploy your own self-hosted Readability clone

 Readability.org is a handy tool, but it also suffers from being useful, popular and free â the service is (understandably) often down, or slow.

 If youâre a heavy Readability user, it might interest you to know that there exist many software libraries...

 Read more »

 March 27, 2016

 Composible Architecture: A real-world example.

 I am the primary maintainer of a significant REST API written using Flask-Restful. Flask-Restful is a pretty full-featured library for creating REST frameworks with Flask. It features everything you need to manage routing to class-based resources, a library for marshalling objects to JSON format, and reqparse, an argparse-inspired library for writing input parsers.

 Read more »

 March 26, 2016

 React.js in the small

 Iâm a big fan of accessible technologies. I get suspicious whenever I encounter a framework or library that is ânot worth it for a small project,â because that is a coded statement that means either âIt has a broad scope of features that a small project wonât take advantage of,â or âit trades off simplicity or user experience for performance or other reasons.â Accessibility also helps open-source projects move from obscure to ubiquitous; if jQuery had required a command-line tool to compile it when it came out, I can practically guarantee you wouldnât have ever heard about it.

 Read more »

 March 19, 2016

 The Web is (finally) a mature platform

 I started my career as a âweb developerâ circa 2007. Iâd dabbled before with Javascript while making sites in HTML/CSS as early as middle school, but that was my first real job as a real software developer. Back in those days, LAMP (Linux/Apache/MySQL/PHP) was about the only sensible way to build a web application. You could use Java servlets, of course, or build applications using CGI, but once PHP was stable and available and straightforward, and it quickly became the obvious choice for a huge range of applications.

 Read more »

 February 28, 2016

 Clojure SQL Migration Libaries

 It is generally recommended, if youâre developing an application that uses a database, that you manage changes to the schema of said database using some sort of migration system or another. Luckily, several intrepid developers in the clojure community have stepped up with their own solutions to this problem â all of which end up being very similar indeed. Read on for an overview of your options and to find out which one is right for you.

 Read more »

 February 20, 2016

 Plotting Reddit post frequencies with d3

 This week I added a better graph to Later (for Reddit)âs post timing analysis page. Previously, the most common time and the most common day were plotted on separate bar charts. However, I noticed that I got called out on this in this article on Medium, which offered a pretty sensible alternative that I wasted no time in ripping off wholesale. I still have the bar graphs in, but I also added a 7x24 color-coded grid plot, allowing for a more accurate view of exactly when the most popular posts were posted.

 Read more »

 Next page
 Page 1 of 7

About Adam

Adam makes a lot of websites. He develops for Telmediq, and is currently working on Later for Reddit, among many others. He is always creating new projects.

 * Twitter: @adambard
 * LinkedIn: adambard
 * Github: adambard

Recent Articles

 1. Jul 26, 2020
 Transitioning Later for Reddit from Docker Machines to Digital Ocean Kubernetes
 2. Dec 17, 2018
 Getting a handle on Reduce
 3. May 23, 2018
 Programatically Liquidating my Steam Inventory
 4. Feb 18, 2018
 One-box stream processing with CSP
 5. Dec 19, 2017
 Running a Side Business as a Programmer: The story of Later for Reddit
 6. Sep 16, 2017
 Javascript FP design pattern: Binding functions

By Year

 1. 2020 1 posts
 2. 2018 3 posts
 3. 2017 7 posts
 4. 2016 12 posts
 5. 2015 13 posts
 6. 2014 28 posts
 7. 2013 28 posts
 8. 2012 22 posts
 9. 2011 10 posts
10. 2010 6 posts

RSS Feed © 2010 - 2022 Adam Bard, Handsome web developer

July 26, 2020
Transitioning Later for Reddit from Docker Machines to Digital Ocean Kubernetes
I recently completed a migration from my old docker-machine-based infrastructure (so 2016!) to Digital Oceanâs hosted kubernetes service.
The bulk of the work was completed in a single evening. It went almost seamlessly, but not so seamlessly that I wasnât able to learn a few things.
The goal of this writeup is not to serve as comprehensive documentation, but hopefully youâll take away some ideas about how to get your
own setup rolling.
Read more »
December 17, 2018
Getting a handle on Reduce
Although traditional algol-inspired languages remain dominant, over the past
decade or so, functional programming techniques utilizing
immutable data structures have seen widespread adoption in a variety of
domains. From React and friends on the frontend to Map-Reduce and the
vast ecosystem of thus-inspired data warehousing systems that power some of
the worldâs largest software applications, the ability to navigate
such applications is now a must-have skill for many programmers.
The subject of todayâs article is a higher-order function called reduce,
also known as fold, aggregate, and others. Reduce is a fundamentally
useful tool, but I find that many developers have trouble grasping and
using it. My hope here is to make an approachable introduction to reduce.
Read more »
May 23, 2018
Programatically Liquidating my Steam Inventory
I donât know about you, but every so often my Steam Inventory gets a bit out of control.
I have no use for Donât Starve trading cards, but itâs not really worth my time to individually
sell each item for a few cents. However, what is worth my time, apparently, is creating a
script to do it for me. This turned out to be quite a winding road, which Iâve documented here.
Read more »
February 18, 2018
One-box stream processing with CSP
If youâre like me (that is, employed by an ad tech company), stream processing is usually
associated with frameworks like Storm,
Flink, Spark Streaming, and other such solutions. However, a
lot of real-life software can be described as stream processing â data comes
in one end, is transformed or aggregated, and goes somewhere else. Many of these
workloads donât justify the overhead of a stream processor, but that doesnât mean
they canât benefit from some of the lessons of stream processing systems.
Read more »
December 19, 2017
Running a Side Business as a Programmer: The story of  Later for Reddit
For the past 5 years, Iâve run a website called
Later for Reddit. For the past 2 years, itâs actually
been making money. Hereâs how that happened.
I used to spend a lot of time on Reddit (insert Mitch Hedberg joke here).
I also used to spend a lot...
Read more »
September 16, 2017
Javascript FP design pattern: Binding functions
Ok, ok, monads have been done. Theyâve even
been done by me before.
But, functional programming is still popular, monads are still very useful for programming
in an FP style, and people are entering the field every day. So hereâs another attempt at
an approachable introduction to usefully apply this pattern to everyday coding problems.
Read more »
September 3, 2017
Bucklescript vs Elm vs Typescript: Typed Javascript showdown!
You, a web developer, have probably heard of Typescript, may have heard of Elm, and
you might even have heard of Bucklescript/ReasonML as well. Each of these
represents a compiles-to-javascript language with strong type support, but each has some different opinions,
philosophies, and features.
In this article Iâll walk you through the features of each of these options and compare their build ecosystems,
editor tooling, and javascript interop. In addition, as is my habit, Iâve written up a small example in each of these languages. (Well, actually,
Iâve taken one of Elmâs small demos and re-created it in the other two). Iâll discuss my impressions
of the tooling and the coding experience along the way.
Read more »
August 28, 2017
OCaml for the impatient
Ocaml is a rad language with some regrettably underdocumented parts. Or, perhaps, overdocumented â
since OCaml attracts people who are interested in rigorous correctness, what documentation exists tends to be rigorous
and correct, and also tedious. This guide will be none of those. Letâs go!
Read more »
August 13, 2017
Best Practices for Javascript Library Versioning
I spend most of my time on this blog talking about Clojure, but IRL I work on
a lot of projects written in a lot of languages. Lately, much of my time has been spent writing and maintaining
React-based frontends in Javascript.
As far as I can tell, no language has solved the problem of dependency management. Some
libraries are better than others, but upgrading dependencies is always an uncomfortable
experience. That being said, Iâve never experienced more pain doing so than in Javascriptâs ecosystem.
Read more »
August 6, 2017
Domain modelling with clojure.spec
 Clojure.spec is, among other things, Clojureâs official answer to tools like Typed Clojure and
Plumaticâs Schema. It represents an attempt to apply some validation to
data and functions, without compromising Clojureâs dynamism and data-is-data philosophy.
In this post, Iâll be working through a sample program by first outlining and modelling
it with the help of clojure.spec, then using spec to guide me while I develop the implementation.
Read more »
March 23, 2017
What programming language should I learn? (2017 edition)
I occasionally find someone asking this question in my inbox. Iâve learned
that for every person that bothers there are a bunch more with the same question,
so I thought Iâd take a break from relentlessly plugging my side hustle
Later for Reddit to answer it publicly.
Read more »
November 4, 2016
On Advice (featuring just one piece of it)
Iâm perfectly comfortable making wild assertions about which
programming technique is better than which other one, but
when it comes to handing out general life or career advice, Iâm a lot stingier
than your average blogger  1 . Even if I was wildly...
Read more »
August 23, 2016
Basic site monitoring with Riemann
 Riemann is a general-purpose event processing system, but its most typical application
is as a place to send and generate metrics about applications. I recently set up a Riemann
server for my personal projects, and I feel like my devops game is stepped up by 1000%.
Or, at very least, I feel like I know know about it as soon as one of my sites goes down.
Read more »
April 15, 2016
What’s in a namespace?
Every well-behaved clojure source file starts with a namespace declaration.
The ns macro, as we all know, is responsible for declaring the namespace
to which the definitions in the rest of the file belong, and generally also
includes some requirements and imports and whatnot. But today (at
Clojure/West, shoutout!) Stuart Sierra made a passing reference to
the internals of ns during his talk that got me interested.
But what is a namespace really?
Read more »
March 29, 2016
Write and deploy your own self-hosted Readability clone
 Readability.org is a handy tool, but it also suffers from being useful, popular and free â
the service is (understandably) often down, or slow.
If youâre a heavy Readability user, it might interest you to know that there exist many
software libraries...
Read more »
March 27, 2016
Composible Architecture: A real-world example.
I am the primary maintainer of a significant REST API written using Flask-Restful.
Flask-Restful is a pretty full-featured library for creating REST frameworks with Flask.
It features everything you need to manage routing to class-based resources, a library
for marshalling objects to JSON format, and reqparse, an argparse-inspired library
for writing input parsers.
Read more »
March 26, 2016
React.js in the small
Iâm a big fan of accessible technologies. I get suspicious whenever I encounter
a framework or library that is ânot worth it for a small project,â because that
is a coded statement that means either âIt has a broad scope of features that
a small project wonât take advantage of,â or âit trades off simplicity or
user experience for performance or other reasons.â Accessibility also helps
open-source projects move from obscure to ubiquitous; if jQuery had required a
command-line tool to compile it when it came out, I can practically guarantee you wouldnât
have ever heard about it.
Read more »
March 19, 2016
The Web is (finally) a mature platform
I started my career as a âweb developerâ circa 2007. Iâd dabbled before
with Javascript while making sites in HTML/CSS as early as middle school,
but that was my first real job as a real software developer. Back
in those days, LAMP (Linux/Apache/MySQL/PHP) was about the only sensible way to
build a web application. You could use Java servlets, of course,
or build applications using CGI, but once PHP was stable and available
and straightforward, and it quickly became the obvious choice for a huge
range of applications.
Read more »
February 28, 2016
Clojure SQL Migration Libaries
It is generally recommended, if youâre developing an application
that uses a database, that you manage changes to the schema of said
database using some sort of migration system or another. Luckily,
several intrepid developers in the clojure community have stepped
up with their own solutions to this problem â all of which end up
being very similar indeed. Read on for an overview of your options
and to find out which one is right for you.
Read more »
February 20, 2016
Plotting Reddit post frequencies with d3
This week I added a better graph to Later (for Reddit)âs
post timing analysis page. Previously,
the most common time and the most common day were plotted on separate bar charts. 
However, I noticed that I got called out on this in this article on Medium,
which offered a pretty sensible alternative that I wasted no time in ripping off
wholesale. I still
have the bar graphs in, but I also added a 7x24 color-coded grid plot, allowing for a more
accurate view of exactly when the most popular posts were posted.
Read more »
Adam Bard
https://adambard.com/blog/
http://dtrace.org/blogs/ahl/
Recent Posts
2016-06-19
None
<div><div>
 	
 	
 	<h2><a href="http://dtrace.org/blogs/ahl/2017/02/12/from-prometheus-to-sisyphus/">From Prometheus to Sisyphus</a></h2>
 
 <p>When Apple announced their new file system, APFS, in June, I hustled to be in the front row of the WWDC presentation, questions with the presenters, and then the open Q&amp;A session. I took a week to write up my notes which turned into as 12 page behemoth of a blog post — longer than my college thesis. Despite <a href="https://twitter.com/ahl/status/743258869523087361">reassurances from the tweeps</a>, I was sure that the blog post was an order of magnitude longer than the modern attention span. I was wrong; so wrong that Ars Technica wanted to republish the blog post. <a href="https://developers.slashdot.org/story/08/01/22/2156244/apple-crippled-its-dtrace-port">Never underestimate the interest in all things Apple</a>.</p>
<p>In that piece I left one big thread dangling. Apple shipped APFS as a technology preview, but they left out access to one of the biggest new features: snapshots. Digging around I noticed that there was a curiously named new system call, “fs_snapshot”, but explicitly didn’t investigate: the post was already too long (I thought), I had spent enough time on it, and someone else (surely! surely?) would want to pull on that thread.</p>
<h4>Slow News Day</h4>
<p>Every so often I’d poke around for APFS news, but there was very little new. Last month folks discovered that <a href="https://arstechnica.com/apple/2017/01/ios-10-3-will-be-apples-first-update-to-convert-storage-to-apfs/">APFS was coming to iOS</a> sooner rather than later. But there wasn’t anything new to play with or any revelations on how APFS would work.</p>
<p>I would search for “APFS snapshots”, “fs_snapshot”, anything I could think of to see if anyone had figured out how to make snapshots work on APFS. Nothing.</p>
<p>A few weekends ago, I decided to yank on that thread myself.</p>
<h4>Prometheus</h4>
<p>I started from the system call, wandered through Apple’s open source kernel, leaned heavily on DTrace, and eventually figured it out. Apple had shipped snapshots in APFS, they just hadn’t made it easy to get there. The folks at Ars were excited for a follow-up, and my investigation turned into this: <a href="https://arstechnica.com/apple/2017/02/testing-out-snapshots-in-apples-next-generation-apfs-file-system/">“Testing out snapshots in Apple’s next-generation APFS file system.”</a></p>
<p>Snapshots were there; the APIs were laid bare; I was going to bring fire to all the Mac fans; <a href="https://arstechnica.com/apple/2015/04/after-fifteen-years-ars-says-goodbye-to-john-siracusas-os-x-reviews/">John Siracusa</a> and <a href="http://www.macworld.com/author/Andy-Ihnatko/">Andy Ihnatko</a> would carry me on their shoulders down the streets of the Internet.</p>
<h4>Sisyphus</h4>
<p>On the eve that this new piece was about to run I was nervously scrolling through Twitter as I took the bus home from <a href="http://www.transposit.com/">work</a>. Now that I had invested the time to research and write-up APFS snapshots I didn’t want someone else beating me to the punch.</p>
<p>Then I found this and my heart dropped:</p>
<blockquote>
<p>Slides from the “Storing our digital lives: Mac filesystems from MFS to APFS” session at MacTech Conference 2016: <a href="https://t.co/uJJuqLL8n9">https://t.co/uJJuqLL8n9</a></p>
<p>— Rich Trouton (@rtrouton) <a href="https://twitter.com/rtrouton/status/799339130643615744">November 17, 2016</a></p></blockquote>
<p>Skim past the craziness of MFS and the hairball of HFS, and start digging through the APFS section. Slide 49, “APFS Snapshots” and there it is “apfs_snapshot” — not a tool that anyone laboriously reverse engineered, deciphering system calls and semi-published APIs — a tool shipped from Apple and included in macOS by default. F — .</p>
<p>Apple had secreted this utility away (along with some others) in <code>/System/Library/Filesystems/apfs.fs/Contents/Resources/</code></p>
<h4>What To Do?</h4>
<p>The article that was initially about a glorious act of discovery had become an article about the reinvention of the wheel. Conversely, vanishingly few people would recognize this as rediscovery since the apfs_snapshot tool was so obscure (9 hits on Google!).</p>
<p>We toned down the already modest chest-thumping and published the article this morning to a pretty nice response so far. I might have happier as an FAKE NEWS Prometheus, blissfully unaware of the pre-existence of fire, but I would have been mortified when the inevitable commenter, one of the few who had used apfs_snapshot, crushed me with my own boulder.</p>
 	
 	
 	<h2><a href="http://dtrace.org/blogs/ahl/2016/12/18/dtrace-at-home/">DTrace at Home</a></h2>
 
 <p>I had been procrastinating making the family holiday card. It was a combination of having a lot on my plate and dreading the formulation of our annual note recapping the year; there were some great moments, but I’m glad I don’t have to do 2016 again. It was just before midnight and either I’d make the card that night or leave an empty space on our friends’ refrigerators. Adobe Illustrator had other ideas:</p>
<p><a href="http://dtrace.org/blogs/ahl/files/2016/12/Screen-Shot-2016-12-17-at-7.03.00-PM.png"></a>
</p>
<p>I’m not the first person to hit this. The problem seems to have existed since CS6 was released in 2016. None of the solutions was working for me, and — inspired by Sara Mauskopf’s <a href="https://medium.com/startup-grind/how-to-start-a-company-with-no-free-time-b70fbe7b918a#.uujdblxc6">excellent post</a> — I was rapidly running out of the time bounds for the project. Enough; I’d just DTrace it.</p>
<p>A colleague scoffed the other day, “I mean, how often do you actually use DTrace?” In his mind DTrace was for big systems, critical system, when dollars and lives were at stake. My reply: I use DTrace every day. I can’t imagine developing software without DTrace, and I use it when my laptop (not infrequently) does something inexplicable (I’m forever grateful to the <a href="http://dtrace.org/blogs/ahl/2006/08/07/dtrace_on_mac_os_x/">Apple team</a> that ported it to Mac OS X).</p>
<p>First I wanted to make sure I had the name of the Illustrator process right:</p>
<pre>$ sudo dtrace -n ‘syscall:::entry{ @[execname] = count(); }’
dtrace: description ‘syscall:::entry’ matched 500 probes
^C
pboard 1
watchdogd 2
awdd 3
...
com.apple.WebKit 7065
Google Chrome He 7128
Google Chrome 8099
Adobe Illustrato 36674</pre>
<p>Glad I checked: “Adobe Illustrato”. Now we can be pretty sure that Illustrator is failing on <code>setrlimit(2)</code> and blowing up as result. Let’s confirm that it is in fact returning -1:</p>
<pre>$ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato"/{ printf("%d %d", arg1, errno); }'
dtrace: description 'syscall::setrlimit:return' matched 1 probe
CPU ID FUNCTION:NAME
 0 532 setrlimit:return -1 1</pre>
<p>There it is. And <code>setrlimit(2)</code> is failing with <code>errno</code> 1 which is <code>EPERM</code> (value too high for non-root user). I already tuned up the files limit pretty high. Let’s confirm that it is in fact setting the files limit and check the value to which it’s being set. To write this script I looked at the documentation for <code>setrlimit(2)</code> (<a href="https://truss.works/blog/2016/12/9/man-splained">hooray for man pages!</a>) to determine that the position of the resource parameter (<code>arg0</code>) and the type of the value parameter (<code>struct rlimit</code>). I needed the DTrace <code>copyin()</code> subroutine to grab the structure from the process’s address space:</p>
<pre>$ sudo dtrace -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this-&gt;r = *(struct rlimit *)copyin(arg1, sizeof (struct rlimit)); printf("%x %x %x", arg0, this-&gt;r.rlim_cur, this-&gt;r.rlim_max); }'
dtrace: description 'syscall::setrlimit:entry' matched 1 probe
CPU ID FUNCTION:NAME
 0 531 setrlimit:entry 1008 2800 7fffffffffffffff</pre>
<p>Looking through <code>/usr/include/sys/resource.h</code> we can see that 1008 corresponds to the number of files (<code>RLIMIT_NOFILE | _RLIMIT_POSIX_FLAG</code>). Illustrator is trying to set that value to 0x7fffffffffffffff or 2⁶³-1. Apparently too big; I filed any latent curiosity for another day.</p>
<p>The quickest solution was to use DTrace again to whack a smaller number into that <code>struct rlimit</code>. Easy:</p>
<pre>$ sudo dtrace -w -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this-&gt;i = (rlim_t *)alloca(sizeof (rlim_t)); *this-&gt;i = 10000; copyout(this-&gt;i, arg1 + sizeof (rlim_t), sizeof (rlim_t)); }'
dtrace: description 'syscall::setrlimit:entry' matched 1 probe
dtrace: could not enable tracing: Permission denied</pre>
<p>Oh right. Thank you <a href="https://en.wikipedia.org/wiki/System_Integrity_Protection">SIP</a>. This isa new laptop (at least a new motherboard due to some bizarre issue) which probably contributed to Illustrator not working when once it did. Because it’s new I haven’t yet disabled the part of SIP that prevents you from using DTrace on the kernel or in destructive mode (e.g. <code>copyout()</code>). <a href="http://internals.exposed/blog/dtrace-vs-sip.html">It’s easy enough to disable</a>, but I’m reboot-phobic — I hate having to restart my terminals — so I went to plan B: <code>lldb</code>.</p>
<p>First I used DTrace to find the code that was calling <code>setrlimit(2)</code>: using some knowledge of the x86 ISA/ABI:</p>
<pre>$ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato" &amp;&amp; arg1 == -1/{ printf("%x", *(uintptr_t *)copyin(uregs[R_RSP], sizeof (uintptr_t)) - 5) }'
dtrace: description 'syscall::setrlimit:return' matched 1 probe
CPU ID FUNCTION:NAME
 0 532 setrlimit:return 1006e5b72
 0 532 setrlimit:return 1006e5b72</pre>
<p>I ran it a few times to confirm the address of the <code>call</code> instruction and to make sure the location wasn’t <a href="https://en.wikipedia.org/wiki/Address_space_layout_randomization#OS_X">being randomized</a>. If I wasn’t in a rush I might have patched the binary, but Apple’s Mach-O Object format always confuses me. Instead I used <code>lldb</code> to replace the call with a store of 0 to <code>%eax</code> (to evince a successful return value) and some <code>nops</code> as padding (hex values I remember due to personal deficiencies):</p>
<pre>(lldb) break set -n _init
Breakpoint 1: 47 locations.
(lldb) run
...
(lldb) di -s 0x1006e5b72 -c 1
0x1006e5b72: callq 0x1011628e0 ; symbol stub for: setrlimit
(lldb) memory write 0x1006e5b72 0x31 0xc0 0x90 0x90 0x90
(lldb) di -s 0x1006e5b72 -c 4
0x1006e5b72: xorl %eax, %eax
0x1006e5b74: nop
0x1006e5b75: nop
0x1006e5b76: nop</pre>
<p>Next I just <code>process detach</code> and got on with making that holiday card…</p>
<h4>DTrace Every Day</h4>
<p>DTrace was designed for solving hard problems on critical systems, but the need to understand how systems behave exists in development and on consumer systems. Just because you didn’t write a program doesn’t mean you can’t fix it.</p>
 	
 	
 	<h2><a href="http://dtrace.org/blogs/ahl/2016/08/09/nomsfs/">A Filesystem on Noms</a></h2>
 
 <p>Since Noms <a href="https://medium.com/@aboodman/noms-init-98b7f0c3566#.ev67mzuzr">dropped last week</a> the dev community has seemed into it. “Git for data” — it simultaneously evokes something very familiar and yet unconstrained. Something that hasn’t been well-noted is how much care the team has taken to make Noms fun to build with, and it is.</p>
<p></p><p>Noms is a content-addressable, decentralized, append-only database. It borrows concepts from a variety of interesting data systems. Obviously databases are represented: Noms is a persistent, transactional data repository. You can also see the fundamentals of git and other decentralized source code management tools. Noms builds up a chain of commits; those chains can be extended, forked, and shared, while historical data are preserved. Noms shares much in common with modern filesystems such as <a href="http://open-zfs.org/wiki/Main_Page">ZFS</a>, <a href="https://btrfs.wiki.kernel.org/index.php/Main_Page">btrfs</a>, or Apple’s forthcoming <a href="http://arstechnica.com/apple/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/">APFS</a>. Like those filesystems, Noms uses copy-on-write, never modifying data <em>in situ</em>; it forms a self-validating hierarchy of data; and it (intrinsically) supports snapshots and efficient movement of data between snapshots.</p>
<p>After learning a bit about Noms I thought it would be interesting to use it as the foundation for a filesystem. I wanted to learn about Noms, and contribute a different sort of example that might push the project in new and valuable ways. The Noms founders, Aaron and Raf, were fired up so I dove in…</p>
<h3>What’s Modern?</h3>
<p>When people talk about a filesystem being “modern” there’s a list of features that they often have in mind. Let’s look at how the Noms database stacks up:</p>
<h4>Snapshots</h4>
<p>A filesystem snapshot preserves the state of the filesystem for some future use — typically data recovery or fast cloning. Since Noms is append-only, every version is preserved. Snapshots are, therefore, a natural side effect. You can make a Noms “snapshot” — any commit in a dataset’s history — writeable by syncing it to a new dataset. Easy.</p>
<h4>Dedup</h4>
<p>The essence of dedup is that unique data should be stored exactly once. If you duplicate a file, a folder, or an entire filesystem the storage consumption should be close to zero. Noms is content addressable, unique data is only ever stored once. Every Noms dataset intrinsically removes duplicated data.</p>
<h4>Consistency</h4>
<p>A feature of a filesystem — arguably <strong>the</strong> feature of a filesystem — is that it shouldn’t ever lose or corrupt your data. One common technique to ensure consistency is to write new data to a new location rather than overwriting old data — so called copy-on-write (COW). Noms is append-only, it doesn’t throw out (or overwrite) old data; copying modified is required and explicit. Noms also recursively checksums all data — a feature of ZFS and btrfs, notably absent from APFS.</p>
<h4>Backup</h4>
<p>The ability to backup your data from a filesystem is almost as important as keeping it intact in the first place. ZFS, for example, lets you efficiently serialize and send the latest changes between systems. When pulling or pushing changes git also efficiently serializes just the changed data. Noms does something similar with its structured data. Data differences are efficiently computed to optimize for minimal data transfer.</p>
<p>Noms has all the core components of a modern filesystem. My goal was to write the translation layer to expose filesystem semantics on top of the Noms interfaces.</p>
<h3>Designing a Schema</h3>
<h4>Initially, Badly</h4>
<p>It’s in the name: Noms eats all the data. Feed it whatever data you like, and let Noms <a href="https://github.com/attic-labs/noms/blob/c6d99f2fabcf870d9cf0cd4de9c540300de3f030/doc/intro.md#type-accretion">infer a schema as you go</a>. For a filesystem though I wanted to define a fixed structure. I started with a schema modeled on a simplified ZFS. Filesystems keep track of files and directories with a structure called an “inode” each of which has a unique integer identifier, the “inode number”. ZFS keeps track of files and directories with DMU objects named by their integer ID. The schema would use a Map&lt;number, Inode&gt; to serve the same function (<strong>spoiler: read on and don’t copy this!</strong>):</p>
<pre>struct Filesystem {
 inodes: Map&lt;Number, struct Inode {
 attr: struct Attr { /* e.g. permissions, modification time, etc. */ }
 contents: Union {
 struct File { data: Ref /* Noms pile of bytes */ } |
 struct Directory { contents: Map&lt;String, Number&gt; }
 }
 }&gt;
 rootInode: Number
 maxInode: Number
}
</pre>
<p>Nice and simple. Files are just Noms Blobs represented by a sequence of bytes. Directories are a Map of strings (the name of the directory entry) to the inode number; the inode number can be used to find the actual content by looking in the top-level map.</p>
<h4>Schema philosophy</h4>
<p>This made sense for a filesystem. Did it make sense for Noms? I wasn’t trying to put the APFS team out of work, rather I was creating a portal from the shell or Finder into Noms. To evaluate the schema, I had the benefit of direct access to the Noms team (and so can all developers at <a href="http://slack.noms.io/">http://slack.noms.io/</a>). I learned two guiding principles for data in Noms:</p>
<p><strong>Noms data should be self-validating. </strong>As much as possible the application should rely on noms to ensure consistency. My schema failed this test because the relationship between inode numbers contained in directories and the entires of the inodes map was something my code alone could maintain and validate.</p>
<p><strong>Noms data should be deterministic.</strong> A given collection of data should have a single representation; the Noms structures should be path and history independent. Two apparently identical datasets should be identical in the eyes of Noms to support efficient storage and transfer, and identical data should produce an identical hash at the root of the dataset. Here, again, my schema fell short because the inode number assigned to a given file or directory depended on how other objects were created. Two different users with two identical filesystems wouldn’t be able to simply sync the way they would with two identical git branches.</p>
<h4>A Better Schema</h4>
<p>My first try made for a fine filesystem, just not a Noms filesystem. With a better understanding of the principles, and with help from the Noms team, I built this schema:</p>
<pre>struct Filesystem {
 root: struct Inode {
 attr: struct Attr { /* e.g. permissions, modification time, etc. */ }
 contents: Union {
 struct File { data: Ref&lt;Blob&gt; /* Noms pile of bytes */ } |
 struct Directory: { contents: Map&lt;string, Cycle&lt;1&gt;&gt; }
 }
 }
}
</pre>
<p>Obviously simpler; the thing that bears explanation is the use of so-called “Cycle” types. A Cycle is a means of expressing a recursive relationship within Noms types. The integer parameter specifies the ancestor struct to which the cycle refers. Consider a linked list type:</p>
<pre>struct LinkedList {
 data: Blob
 next: Cycle&lt;0&gt;
}</pre>
<p>The “next” field refers to immediately containing struct, LinkedList. In our filesystem schema, a Directory’s contents are represented by a map of strings (directory entry names) to Inodes, Cycle&lt;1&gt; referring to the struct “above” or “containing” the Directory type. (Read on for a visualization of this.)</p>
<h3>Writing It</h3>
<p>To build the filesystem I picked a <a href="https://github.com/hanwen/go-fuse">FUSE binding for Go</a>, dug into the Noms APIs, and <a href="http://dtrace.org/blogs/ahl/2016/08/02/i-love-go-i-hate-go/">wrestled my way through some Go heartache</a>.</p>
<p>Working with Noms requires a slightly different mindset than other data stores. Recall in particular that Noms data is immutable. Adding a new entry into a Map creates a new Map. Setting a member of a Struct creates a new Struct. Changing nested structures such as the one defined by our schema requires unzipping it, and then zipping it back together. Here’s a Go snippet that demonstrates that methodology for creating a new directory:</p>
<h3>Demo</h3>
<p>Showing it off has all the normal glory of a systems demo! Check out the <a href="https://github.com/attic-labs/noms/blob/master/samples/go/nomsfs/README.md">documentation</a> for requirements.</p>
<p>Create and mount a filesystem from a new local Noms dataset:</p>
<pre>$ go build
$ mkdir /var/tmp/mnt
$ go run nomsfs.go /var/tmp/nomsfs::fs /var/tmp/mnt
running...</pre>
<p>You can open the folder and drop data into it.</p>
<div><p><a href="http://dtrace.org/blogs/ahl/files/2016/08/Screen-Shot-2016-08-06-at-3.36.24-PM.png"></a></p><p>Your database fell into my filesystem!</p></div>
<p>Now let’s take a look at the underlying Noms dataset:</p>
<pre>$ noms show http://demo.noms.io/ahl_blog::fs
struct Commit {
 meta: struct {},
 parents: Set&lt;Ref&lt;Cycle&lt;0&gt;&gt;&gt;,
 value: struct Filesystem {
 root: struct Inode {
 attr: struct Attr {
 ctime: Number,
 gid: Number,
 mode: Number,
 mtime: Number,
 uid: Number,
 xattr: Map&lt;String, Blob&gt;,
 },
 contents: struct Directory {
 entries: Map&lt;String, Cycle&lt;1&gt;&gt;,
 } | struct Symlink {
 targetPath: String,
 } | struct File {
 data: Ref&lt;Blob&gt;,
 },
 },
 },
}({
 meta: {},
 parents: {
 5v82rie0be68915n1q7pmcdi54i9tmgs,
 },
 value: Filesystem {
 root: Inode {
 attr: Attr {
 ctime: 1.4705227450393803e+09,
 gid: 502,
 mode: 511,
 mtime: 1.4705227450393803e+09,
 uid: 110853,
 xattr: {},
 },
 contents: Directory {
 entries: {
 "usenix_winter91_faulkner.pdf": Inode {
 attr: Attr {
 ctime: 1.4705228859273868e+09,
 gid: 502,
 mode: 420,
 mtime: 1.468425783e+09,
 uid: 110853,
 xattr: {
 "com.apple.FinderInfo": 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 // 32 B
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00,
 "com.apple.quarantine": 30 30 30 31 3b 35 37 38 36 36 36 33 37 3b 53 61 // 21 B
 66 61 72 69 3b,
 },
 },
 contents: File {
 data: dmc45152ie46mn3ls92vvhnm41ianehn,
 },
 },
 },
 },
 },
 },
})</pre>
<p>You can see the type at the top and then the actual filesystem contents. Let’s look at more complicated example where I’ve taken <a href="http://inception.davepedu.com/inception.mp3">part of the Noms source tree and copied it to nomsfs</a>. You can navigate around its structure courtesy of the Splore utility (take particular note of nested directories that show the recursive data definition described above):</p>
<p><small>Embedded ‘Splore! <a href="http://splore.noms.io/?db=https://demo.noms.io/ahl_blog&amp;hash=2nhi5utm4s38hka22vt9ilv5i3l8r2ol">http://splore.noms.io/?db=https://demo.noms.io/ahl_blog&amp;hash=2nhi5utm4s38hka22vt9ilv5i3l8r2ol</a></small><br />
</p>
<p>You can see the all of the various states that the filesystem has been through — each state change — using noms log http://demo.noms.io/ahl_blog::fsnoms. You can sync it to your local computer with noms sync http://demo.noms.io/ahl_blog::fsnoms /var/tmp/fs or checkout some previous state from the log (just like a filesystem snapshot). Diff two states from the log or make your own changes and diff it with the original using noms diff.</p>
<h3>Nom Nom Nom</h3>
<p>It took less than 1000 lines of Go code to make Noms appear as a Window in the Finder, eating data as quickly as I could drag and drop (<a href="https://github.com/attic-labs/noms/tree/master/samples/go/nomsfs">try it!</a>). Imagine what Noms might look like behind other known data interfaces; it could bring git semantics to existing islands of data. Noms could form the basis of a new type of <a href="http://martinfowler.com/bliki/DataLake.html">data lake</a> — maybe one that’s simple and powerful enough to bring real results. Beyond the marquee features, Noms is fun to build with. I’m already working on my next Noms application.</p>
 	
 	
 	<h2><a href="http://dtrace.org/blogs/ahl/2016/08/02/i-love-go-i-hate-go/">I Love Go; I Hate Go</a></h2>
 
 <p>I liked Go right away. It was close enough to C and Java to be instantly familiar, the examples and tutorials were straightforward, and I was quickly writing real code. I’ve wanted to learn Go since its <a href="http://redmonk.com/sogrady/2016/07/20/language-rankings-6-16/#advanced_iframe">popularity was surging</a> few years ago. In no danger of being judged an early adopter, I happily found a great project that—as it happened—had to be in Go (more in a future post).</p>
<h3>I Love Go</h3>
<p>My first priority was not looking stupid. The folks I’d be doing this for are actual Go developers; I wanted my code to fit in without imposing with too many questions. They had no style guide. Knowing that my 80-column width sensibilities expose an unattractive nostalgia, I went looking for a max line length. There wasn’t one. Then I discovered gofmt, the simple tool that Go employs to liberate developers from the tyranny of stylistic choice. It takes Go code and spits it back out in the One True Style. I love it. I was raised in an engineering culture with an <a href="https://www.cis.upenn.edu/~lee/06cse480/data/cstyle.ms.pdf">exacting style guide</a>, but any style guide has gaps. Factions form; style-originalists face off against those who view (incorrectly!) the guide as a living document. I updated my decades-old <a href="https://github.com/ahl/dotfiles/blob/master/.vimrc">.vimrc</a> to gofmt on save. These Go tyrants were feeling like my kind of tyrant.</p>
<p>One of the things that turned me off of C++ (98, 11, and 14) is its increasing amount of magic. Go is comparatively straightforward. When I reach for something I find that it’s where I expect it to be. Libraries and examples aren’t mysterious. Error messages are non-mysterious (other than quickly resolved confusion about methods with “<a href="http://stackoverflow.com/questions/33936081/golang-method-with-pointer-receiver">pointer receivers</a>”). Contrast this with <a href="http://dtrace.org/blogs/ahl/2015/06/22/first-rust-program-pain/">Rust whose errors read like mind-bindingly inscrutable tax forms</a>.</p>
<p>Go’s containment-based inheritance is easy enough to reason about. Its interfaces are similarly no-nonsense and pragmatic. You don’t have to define a structure as implementing an interface. You can use an interface to describe anything that implements it. Simple. This is particularly useful for testing. In a messy environment with components beyond your control you can apply interfaces to other people’s code, and then mock it out.</p>
<p>The toolchain is, again, simple to use—the benefit of starting from scratch—and makes for fast compilation, quick testing, and easy integration with a good-sized ecosystem. I stopped worrying about dependencies, rebuilding, etc. knowing that go run would find errors wherever I had introduced them and do so remarkably quickly.</p>
<h3>I Hate Go</h3>
<p>Go is opinionated. Most successful products have that strong sense of what they are and what they aren’t; Go draws as sharp a line as any language. I was seduced by its rightheadedness around style, but with anything or anyone that opinionated you’ll find some of those opinions weird and others simply off-putting.</p>
<p>Reading the official documentation, I found myself in the middle of a section prefixed with the phrase “if GOOS is set to plan9”. Wow. I’m a few standard deviations from the norm in terms of being an OS nerd, but I’ve never even seen Plan 9. I knew that the Plan 9 folks got the band back together to create Go; it’s great that their pop audiences don’t dissuade them from playing off their old B-sides. Quirky, but there’s nothing wrong with that.</p>
<p>I wanted to check an invariant; how does Go do assertions? <a href="https://golang.org/doc/faq#assertions">Fuck you, you’re a bad programmer.</a> That’s how. The Go authors feel so strongly about asserts being typically misused that they refuse to provide them. So folks use one (or more) of some workable libraries.</p>
<p>I created an infinite recursion, overflowing the stack. Go produces the <a href="https://golang.org/src/runtime/traceback.go#L581">first 100 stack frames and that’s it</a>. Maybe you can change that, but I couldn’t figure out how. (“go stackoverflow” is about the most useless thing you can search for; chapeau, <a href="https://twitter.com/ahl/status/753254488123727873">Go</a> and Stackoverflow respectively.) I could be convinced that I only want 100 stack frames, but not the last 100, not the same 4 over and over again. I ended up limiting the stack size with <a href="https://golang.org/pkg/runtime/debug/#SetMaxStack">runtime.debug.SetMaxStack()</a>, Goldilocks-ing it between too big to catch the relevant frames and too small to allow for normal operation.</p>
<p>I tried using other tools (ahem, DTrace) to print the stack, but, of course, the Go compiler omits frame pointers rendering the stacks unobservable to debuggers. Ditto arguments due to ABI-non-compliant calling conventions, but that’s an aside. The environment variable GOEXPERIMENT=framepointer is supposed to compile with frame pointers, but it was a challenge to rebuild the world. All paths seemed to lead me to my former-colleague’s scathing synopsis: <a href="http://dtrace.org/blogs/wesolows/2014/12/29/golang-is-trash/">Golang is Trash</a>.</p>
<p>As fun as it is to write code in Go, debugging in Go is no fun at all. I may well just be ignorant of the right tooling. But there sure isn’t a debugger with the simple charm of go build for compilation, go test for testing, or go run for execution.</p>
<h3>Immaturity and Promise</h3>
<p>Have you ever been in a relationship where minor disagreements immediately escalated to “maybe we should break up?” The Go documentation even seems ready to push you to some other language at the slightest affront. Could I have asserts? Sure, if you’re a bad programmer. Perhaps ABI compliance has its merits? I’m sure you could find that in some other language. Could you give me the absolute value of this int? Is something wrong with your ‘less than’ key?</p>
<p>I’m sure time will, as it tends to, bring pragmatism. I appreciate that Go does have strong opinions (it’s just hard to remember that when I disagree with them). Weak opinions are what turn languages into unreadable mishmashes of overlapping mechanism. My favorite example of this is Perl. My first real programming job was in Perl. I was the most avid reader teenage of the Perl llama and camel books. Ask me about my chat with Larry Wall, Perl’s creator, if you see a beer in my hand. <a href="https://www.amazon.com/Masterminds-Programming-Conversations-Creators-Languages/dp/0596515170">In an interview</a>, Larry said, “In Perl 6, we decided it would be better to fix the language than fix the user”. Contrast this with <a href="https://golang.org/doc/faq#assertions">Go’s statement on assertions</a>:</p>
<blockquote><p>Go doesn’t provide assertions. They are undeniably convenient, but our experience has been that programmers use them as a crutch to avoid thinking about proper error handling and reporting.</p></blockquote>
<p>Perl wants to be whatever the user wants to be. The consumate pleaser. Go sees the user as flawed and the strictures of the language as the cure. It’s an authoritarian, steadfast in its ideals, yet too sensitive to find compromise (sound like anyone we all know?).</p>
<p>Despite my frustrations I really enjoy writing code in Go. It’s clean and supported by a great community and ecosystem. I’m particularly heartened that <a href="https://github.com/golang/go/issues/15840">Go 1.7 will compile with frame pointers by default</a>. Diagnosing certain types of bugs is still a pain in the neck, but I’m sure it will improve and I’ll see where I can pitch in.</p>
 	
 	
 	<h2><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part6/">APFS in Detail: Conclusions</a></h2>
 
 <p><em>This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1">first post</a> for the table of contents.</em></p>
<h2>Summing Up</h2>
<p>I’m not sure Apple absolutely had to replace HFS+, but likely they had passed an inflection point where continuing to maintain and evolve the 30+ year old software was more expensive than building something new. APFS is a product born of that assessment.</p>
<p>Based on what Apple has shown I’d surmise that its core design goals were:</p>
<ul>
<li>satisfying all consumers (laptop, phone, watch, etc.)</li>
<li>encryption as a first-class citizen</li>
<li>snapshots for modernized backup.</li>
</ul>
<p>Those are great goals that will benefit all Apple users, and based on the WWDC demos APFS seems to be on track (though the macOS Sierra beta isn’t quite as far along).</p>
<p>In the process of implementing a new file system the APFS team has added some expected features. HFS was built when 400KB floppies ruled the Earth (recognized now only as the ubiquitous and anachronistic save icon). Any file system started in 2014 should of course consider huge devices, and SSDs–check and check. Copy-on-write (COW) snapshots are the norm; making the Duplicate command in the Finder faster wasn’t much of a detour. The use case is unclear, it’s a classic <a href="http://faculty.babson.edu/krollag/org_site/encyclop/garbage_can.html">garbage can theory</a> solution, a solution in search of a problem, but it doesn’t hurt and it makes for a fun demo. The beach ball of doom earned its nickname; APFS was naturally built to avoid it.</p>
<p>There are some seemingly absent or ancillary design goals: performance, openness, and data integrity. Squeezing the most IOPS or throughput out of a device probably isn’t critical on watchOS, and it’s relevant only to a small percentage of macOS users. It will be interesting to see how APFS performs once it ships (measuring any earlier would only misinform the public and insult the APFS team). The APFS development docs have a bullet on open source: <a href="https://developer.apple.com/library/prerelease/content/documentation/FileManagement/Conceptual/APFS_Guide/UsingtheAppleFileSystem/UsingtheAppleFileSystem.html#//apple_ref/doc/uid/TP40016999-CH4-DontLinkElementID_23">“An open source implementation is not available at this time.”</a> I don’t expect APFS to be open source at this time or any other, but prove me wrong, Apple. If APFS becomes world-class I’d love to see it in Linux and FreeBSD–maybe Microsoft would even jettison their ReFS experiment. My experience with OpenZFS has shown that open source accelerates that path to excellence. It’s a shame that APFS lacks checksums for user data and doesn’t provide for data redundancy. Data integrity should be job one for a file system, and I believe that that’s true for a watch or phone as much as it is for a server.</p>
<p>At stability, APFS will be an improvement, for Apple users of all kinds, on every device. There are some clear wins and some missed opportunities. Now that APFS has been shared with the world the development team is probably listening. While Apple is clearly years past the decision to build from scratch rather than adopting existing modern technology, there’s time to raise the priority of data integrity and openness. I’m impressed by Apple’s goal of using APFS by default within 18 months. Regardless of how it goes, it will be an exciting transition.</p>
 	
 	
 	<h2><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/">APFS in Detail: Data Integrity</a></h2>
 
 <p><em>This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1">first post</a> for the table of contents.</em></p>
<h2>Data Integrity</h2>
<p>Arguably the most important job of a file system is preserving data integrity. Here’s my data, don’t lose it, don’t change it. If file systems could be trusted absolutely then the “only” reason for backup would be the idiot operators (i.e. you and me). There are a few mechanisms that file systems employ to keep data safe.<br />
<a></a></p>
<h3>Redundancy</h3>
<p>APFS makes no claims with regard to data redundancy. As Apple’s Eric Tamura noted at WWDC, most Apple devices have a single storage device (i.e. one logical SSD) making RAID, for example, moot. Instead redundancy comes from lower layers such as Apple RAID (apparently a thing), hardware RAID controllers, SANs, or even the “single” storage devices themselves..</p>
<p>As an aside note that SSDs in most Apple products where APFS will run include <a href="https://www.ifixit.com/Teardown/MacBook+Pro+13-Inch+Retina+Display+Early+2015+Teardown/38300#s86949">multiple more-or-less independent NAND chips</a>. High-end SSDs do implement data redundancy within the device, but it comes at the price of reduced capacity and performance. As noted above, the “flash-optimization” of APFS doesn’t actually extend much below the surface of the standard block device interface, but the raw materials for innovation are there.</p>
<p>Also, APFS removes the most common way of a user achieving local data redundancy: copying files. A copied file in APFS actually creates a lightweight clone with no duplicated data. Corruption of the underlying device would mean that both “copies” were damaged whereas with full copies localized data corruption would affect just one.<br />
<a></a></p>
<h3>Crash Consistency</h3>
<p>Computer systems can fail at any time—crashes, bugs, power outages, etc.—so file systems need to anticipate and recover from these scenarios. The old old old school method is to plod along and then have a special utility to check and repair the file system during boot (fsck, short for file system check). More modern systems labor to achieve an always consistent format, or only narrow windows of inconsistency, obviating the need for the full, expensive fsck. ZFS, for example, builds up new state on disk and then atomically transitions from the previous state to the new one with a single atomic operation.</p>
<p>Overwriting data creates the most obvious opening for inconsistency. If the file system needs to overwrite several regions there is a window where some regions represent the new state and some represent the former state. Copy-on-write (COW) is a method to avoid this by always allocating new regions and then releasing old ones for reuse rather than modifying data in-place. APFS claims to implement a “novel copy-on-write metadata scheme”; APFS lead developer Dominic Giampaolo emphasized the novelty of this approach without delving into the details. In conversation later, he made it clear that APFS does not employ the ZFS mechanism of copying all metadata above changed user data which allows for a single, atomic update of the file system structure.</p>
<p>It’s surprising to see that APFS includes <a href="https://forums.developer.apple.com/thread/49207">fsck_apfs</a>—even after asking Dominic I’m not sure why it would be necessary. For comparison I don’t believe there’s been an instance where fsck for ZFS would have found a problem that the file system itself didn’t already know how to detect. But Dominic was just as confused about why ZFS would forego fsck, so perhaps it’s just a matter of opinion.<br />
<a></a></p>
<h3>Checksums</h3>
<p>Notably absent from the APFS intro talk was any mention of checksums. A checksum is a digest or summary of data used to detect (and correct) data errors. The story here is surprisingly nuanced. APFS checksums its own metadata but not user data. The justification for checksumming metadata is strong: there’s relatively not much of it (so the checksums don’t consume much storage) and losing metadata can cast a potentially huge shadow of data loss. If, for example, metadata for a top level directory is corrupted then potentially all data on the disk could be rendered inaccessible. ZFS duplicates metadata (and triple duplicates top-level metadata) for exactly this reason.</p>
<p>Explicitly not checksumming user data is a little more interesting. The APFS engineers I talked to cited strong ECC protection within Apple storage devices. Both flash SSDs and magnetic media HDDs use redundant data to detect and correct errors. The engineers contend that Apple devices basically don’t return bogus data. NAND uses extra data, e.g. 128 bytes per 4KB page, so that errors can be corrected and detected. (For reference, ZFS uses a fixed size 32 byte checksum for blocks ranging from 512 bytes to megabytes. That’s small by comparison, but bear in mind that the SSD’s ECC is required for the expected analog variances within the media.) The devices have a bit error rate that’s tiny enough to expect no errors over the device’s lifetime. In addition, there are other sources of device errors where a file system’s redundant check could be invaluable. SSDs have a multitude of components, and in volume consumer products they rarely contain end-to-end ECC protection leaving the possibility of data being corrupted in transit. Further, their complex firmware can (does) contain bugs that can result in data loss.</p>
<p>The Apple folks were quite interested in my experience with regard to bit rot (aging data silently losing integrity) and other device errors. I’ve seen many instances where devices raised no error but ZFS (correctly) detected corrupted data. Apple has some of the most stringent device qualification tests for its vendors; I trust that they really do procure the best components. Apple engineers I spoke with claimed that bit rot was not a problem for users of their devices, but if your software can’t detect errors then you have no idea how your devices really perform in the field. ZFS has found data corruption on multi-million dollar storage arrays; I would be surprised if it didn’t find errors coming from TLC (i.e. the cheapest) NAND chips in some of Apple’s devices. Recall the (fairly) recent <a href="http://www.iphonehacks.com/2014/11/128gb-iphone-6-and-plus-boot-loops.html">brouhaha regarding storage problems in the high capacity iPhone 6</a>. At least some of Apple’s devices have been imperfect.</p>
<p>As someone who has data he cares about on a Mac, who has seen data lost from HFS, and who knows that even expensive, enterprise-grade equipment can lose data, I would gladly sacrifice 16 bytes per 4KB–less than 1% of my device’s size.<br />
<a></a></p>
<h3>Scrub</h3>
<p>As data ages you might occasionally want to check for bit rot. Likely fsck_apfs can accomplish this; as noted though there’s no data redundancy and no checksums for user data, so scrub would only help to find problems and likely wouldn’t help to correct them. And if it makes it any easier for Apple to reverse course, let’s say it’s for the el cheap-o drive I bought from Fry’s not for the gold-plated device I got from Apple.</p>
<p><em>Next in this series: <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part6/">Conclusions</a></em></p>
 	
 	
 	<h2><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part4/">APFS in Detail: Performance</a></h2>
 
 <p><em>This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1">first post</a> for the table of contents.</em></p>
<h2>Performance</h2>
<p>APFS claims to be optimized for flash. Flash memory (NAND) is the stuff in your speedy SSD. Apple changed the computing industry when it put flash into the iPod and iPhone, volumes for which fundamentally changed the economics of flash. This consumer change impacted the enterprise (as it often does), giving rise to <a href="http://dtrace.org/blogs/ahl/2008/11/10/hybrid-storage-pools-in-the-7410/">hybrid</a> and <a href="https://techcrunch.com/2015/11/19/how-pure-storage-took-a-different-approach-to-storage/">all-flash arrays</a>. <a href="http://www.storagesearch.com/ssd-ram-flash%20pricing.html">Ten years ago flash cost as much as DRAM</a>; now it’s challenging the economics of hard disks.</p>
<p>SSDs mimic the block interface of conventional hard drives, but the underlying technology is completely different. In particular while magnetic media can read or write sectors arbitrarily, flash erases large chunks (blocks) and reads and writes smaller chunks (pages). The management is done by what’s called the flash translation layer (FTL), software that makes blocks and pages appear more like a hard drive. An FTL is very similar to a file system, creating a virtual mapping (a translation) between block addresses and locations within the media. Apple controls the full stack including the SSD, FTL, and file system; they could have built something differentiated, <a href="http://queue.acm.org/detail.cfm?id=2463636">optimizing this components to work together</a>. What APFS does, however, is simply write in patterns known to be more easily handled by NAND. It’s a file system with flash-aware characteristics rather than one written explicitly for the native flash interfaces, more or less what you’d expect in 2016.</p>
<p>Also on the topic of flash, APFS includes TRIM support. TRIM is a command in the ATA protocol that allows a file system to indicate to an SSD (specifically, its FTL) that some space has been freed. SSDs require significant free space and perform better when there’s more of it; they include more physical space than they advertise. For example, my 1TB SSD includes 1TB (2<sup>40</sup> = 1024<sup>4</sup>) bytes of flash but only reports 931GB of available space, sneakily matching the storage industry’s self-serving definition of 1TB (1000<sup>4</sup> = 1 trillion bytes). With more free space, FTLs can trade off space efficiency for performance and longevity. TRIM has become expected of file systems; it’s unsurprising that APFS supports it. The problem with TRIM though is that it’s only useful when there’s free space: it’s something of a benchmark special. Once your disk is mostly full (as mine are in my laptop and phone basically at all times) TRIM doesn’t do anything for you. I doubt that TRIM will bring any discernible benefit for APFS users beyond the placebo effect of feature parity.</p>
<p>APFS also focuses on latency; Apple’s number one goal is to avoid the beachball of doom. APFS addresses this with I/O QoS (quality of service) to prioritize accesses that are immediately visible to the user over background activity that doesn’t have the same time-constraints. This is inarguably a benefit to users and a sophisticated file system capability.</p>
<p><em>Next in this series: <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/">Data Integrity</a></em></p>
 	
 	
 	<h2><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part3/">APFS in Detail: Space Efficiency and Clones</a></h2>
 
 <p><em>This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1">first post</a> for the table of contents.</em><br />
<a></a></p>
<h2>Space Efficiency</h2>
<p>A modern trend in file systems has been to store data more efficiently to effectively increase the size of your device. Common approaches include compression (which, as noted above, is very very likely coming) and deduplication. Dedup finds common blocks and avoids storing them multiply. This is potentially highly beneficial for file servers where many users or many virtual machines might have copies of the same file; it’s probably not useful for the single-user or few-user environments that Apple cares about. (Yes, they have <a href="http://www.apple.com/osx/server/features/#xsan">server-ish offerings</a> but their heart clearly isn’t into it.) It’s also furiously hard to do well as I learned painfully while supporting ZFS.</p>
<p>Apple’s sort-of-unique contribution to space efficiency is constant time cloning of files and directories. As a quick aside, “files” in macOS are often really directories; it’s a convenient lie they tell to allow logically related collections of files to be treated as an indivisible unit. Right click an application and select “Show Package Contents” to see what I mean. Accordingly, I’m going to use the term “file” rather than “file or directory” in sympathy for the patient readers who have made it this far.</p>
<p>With APFS, if you copy a file within the same file system (or possibly the same container; more on this later), no data is actually duplicated. Instead a constant amount of metadata is updated and the on-disk data is shared. Changes to either copy cause new space to be allocated (so-called “copy on write” or COW).</p>
<p>I haven’t seen this offered in other file systems, and it clearly makes for a good demo, but it got me wondering about the use case (<strong>UPDATE</strong>: btrfs supports this and calls the feature <a href="https://lwn.net/Articles/331808/">“reflinks”</a>–link by reference). Copying files between devices (e.g. to a USB stick for sharing) still takes time proportional to the amount of data copied of course. Why would I want to copy a file locally? The common case I could think of is the layman’s version control: “thesis”, “thesis-backup”, “thesis-old”, “thesis-saving because I’m making edits while drunk”.</p>
<p>There are basically three categories of files:</p>
<ul>
<li>Files that are fully overwritten each time; images, MS Office docs, videos, etc.</li>
<li>Files that are appended to, mostly log files</li>
<li>Files with a record-based structure, such as database files.</li>
</ul>
<p>For the average user, most files fall into that first category. So with APFS I can make a copy of my document and get the benefits of space sharing, but those benefits will be eradicated as soon as I save the new revision. Perhaps users of larger files have a greater need for this and have a better idea of how it might be used.</p>
<p>Personally, my only use case is taking a file, say time-shifted Game of Thrones episodes falling into the “fair use” section of copyright law, and sticking it in Dropbox. Currently I need to choose to make a copy or permanently move the file to my Dropbox folder. Clones would let me do this more easily. But then so would hard links (a nearly ubiquitous file system feature that lets a file appear in multiple directories).</p>
<p>Clones open the door for potential confusion. While copying a file may take up no space, so too deleting a file may free no space. Imagine trying to free space on your system, and needing to hunt down the last clone of a large file to actually get your space back.</p>
<p><a href="http://dtrace.org/blogs/ahl/files/2016/06/screenshot853.jpg"></a>APFS engineers don’t seem to have many use cases in mind; at WWDC they asked for suggestions from the assembled developers (the best I’ve heard is for copied VMs; not exactly a mass-market problem). If the focus is generic revision control, I’m surprised that Apple didn’t shoot for a more elegant solution. One could imagine functionality with APFS that allows a user to enable per-file Time Machine, change tracking for any file. This would create a new type of file where each version is recorded transparently and automatically. You could navigate to previous versions, prune the history, or delete the whole pile of versions at once (with no stray clones to hunt down). In fact, Apple <a href="http://arstechnica.com/apple/2011/07/mac-os-x-10-7/14/#versioning-internals">introduced something related 5 years ago</a>, but I’ve literally never seen or heard of it until researching this post (show of hands if you’ve clicked “Browse All Versions…”). APFS could clean up its implementation, simplify its use, and bring generic support for all applications. None of this solves my Game of Thrones storage problem, but I’m not even sure it’s much of a problem…</p>
<p>Side note: Finder copy creates space-efficient clones, but cp from the command line does not.</p>
<p><em>Next in this series: <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part4/">Performance</a></em></p>
 	
 	
 	<h2><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part2/">APFS in Detail: Encryption, Snapshots, and Backup</a></h2>
 
 <p><em>This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1">first post</a> for the table of contents.</em><br />
<a></a></p>
<h2>Encryption</h2>
<p>Encryption is clearly a core feature of APFS. This comes from diverse requirements from the various devices, for example multiple keys within file systems on the iPhone or per-user keys on laptops. I heard the term “innovative” quite a bit at WWDC, but here the term is aptly applied to APFS. It supports several different encryption choices for a file system:</p>
<ul>
<li>Unencrypted</li>
<li>Single-key for metadata and user data</li>
<li>Multi-key with different choices for metadata, files, and even sections of a file (“extents”)</li>
</ul>
<p>Multi-key encryption is particularly relevant for portables where all data might be encrypted, but unlocking your phone provides access to an additional key and therefore additional data. Unfortunately this doesn’t seem to be working in the first beta of macOS Sierra (specifying fileEncryption when creating a new volume with diskutil results in a file system that reports “Is Encrypted” as “No”).</p>
<p>Related to encryption, I noticed an undocumented feature while playing around with diskutil (which prompts you for interactive confirmation of the destructive power of APFS unless this is added to the command-line: -IHaveBeenWarnedThatAPFSIsPreReleaseAndThatIMayLoseData; I’m not making this up). APFS (apparently) supports constant time cryptographic file system erase, called “effaceable” in the diskutil output. This presumably builds a secret key that cannot be extracted from APFS and encrypts the file system with it. A secure erase then need only delete the key rather than needing to scramble and re-scramble the full disk to ensure total eradication. Various iOS docs refer to this capability requiring some specialized hardware; it will be interesting to see what the option means on macOS. Either way, let’s not mention this to the FBI or NSA, agreed?<br />
<a></a></p>
<h2>Snapshots and Backup</h2>
<p>APFS brings a much-desired file system feature: snapshots. A snapshot lets you freeze the state of a file system at a particular moment and continue to use and modify that file system while preserving the old data. It does so in a space-efficient fashion where, effectively, changes are tracked and only new data takes up additional space. This has the potential to be extremely valuable for backup by efficiently tracking the data that has changed since the last backup.</p>
<p>ZFS includes snapshots and serialization mechanisms that make it efficient to backup file systems or transfer file systems to a remote location. Will APFS work like that? Probably not, answered Dominic Giampaolo, APFS lead developer. ZFS sends all changed data while Time Machine can have exclusion lists and the like. That seems surmountable, but we’ll see what Apple does. APFS right now is incompatible with Time Machine due to the lack of directory hard links, a fairly disgusting implementation that likely contributes to Time Machine’s questionable reliability. Hopefully APFS will create some efficient serialization for Time Machine backup.</p>
<p>While Eric Tamura, APFS dev manager, demonstrated snapshots at WWDC, the required utilities aren’t included in the macOS Sierra beta. I used DTrace (technology I’m increasingly amazed that Apple ported from OpenSolaris) to find a tantalizingly-named new system call fs_snapshot; I’ll leave it to others to reverse engineer its proper use.<br />
<a></a></p>
<h2>Management</h2>
<p>APFS brings another new feature known as space sharing. A single APFS “container” that spans a device can have multiple “volumes” (file systems) within it. Apple contrasts this with the static allocation of disk space to support multiple HFS+ instances, which seems both specious and an uncommon use case. Both ZFS and btrfs have a similar concept of a shared pool of storage with nested file systems for administration and management.</p>
<p>Speaking with Dominic and other members of the APFS team, we discussed how volumes are the unit by which users can control things like snapshots and encryption. You’d want multiple volumes to correspond with different policies around those settings. For example while you might want to snapshot and backup your system each day, the massive /private/var/vm/sleepimage (for saving memory when hibernating) should live on its own and not be backed up.</p>
<p>Space sharing is more like an operational detail than a game changing feature. You can think of it like special folders with snapshot and encryption controls… which is probably why Apple’s marketing department has yet to make me a job offer. Unfortunately this feature isn’t working in the macOS Sierra beta, so I wasn’t able to have more than one volume per container. Adding new volumes can fail with an opaque error (-69625 mean anything to you?), but using a larger disk image resolve the problem.</p>
<p><em>Next in this series: <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part3/">Space Efficiency and Clones</a></em></p>
 	
 	
 	<h2><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/">APFS in Detail: Overview</a></h2>
 
 <p><a href="http://dtrace.org/blogs/ahl/files/2016/06/wwdc16-og.jpg"></a>Apple announced a new file system that will make its way into all of its OS variants (macOS, tvOS, iOS, watchOS) in the coming years. Media coverage to this point has been mostly breathless elongations of <a href="https://developer.apple.com/library/prerelease/content/documentation/FileManagement/Conceptual/APFS_Guide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40016999-CH1-DontLinkElementID_27">Apple’s developer documentation</a>. With a dearth of detail I decided to attend the <a href="http://devstreaming.apple.com/videos/wwdc/2016/701q0pnn0ietcautcrv/701/701_introducing_apple_file_system.pdf">presentation</a> and Q&amp;A with the APFS team at WWDC. Dominic Giampaolo and Eric Tamura, two members of the APFS team, <a href="https://developer.apple.com/videos/play/wwdc2016/701/">gave an overview to a packed room</a>; along with other members of the team, they patiently answered questions later in the day. With those data points and some first hand usage I wanted to provide an overview and analysis both as a user of Apple-ecosystem products and as a long-time operating system and file system developer.</p>
<p>I’ve divided my review into several sections that span a few posts. I’d encourage you to jump around to topics of interest or skip right to <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part6/#apfs-conclusion">the conclusion</a> (or to the <a href="https://twitter.com/ahl/status/743923994466758657">tweet summary</a>). Highest praise goes to <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part2/#apfs-encryption">encryption</a>; ire to <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/#apfs-data">data integrity</a>.</p>
<ul>
<li><a href="#apfs-basics">Basics</a></li>
<li><a href="#apfs-debt">Paying Down Debt</a></li>
<li><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part2/#apfs-encryption">Encryption</a></li>
<li><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part2/#apfs-snapshots">Snapshots and Backup</a></li>
<li><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part2/#apfs-management">Management and Space Sharing</a></li>
<li><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part3/#apfs-clones">Space Efficiency and Clones</a></li>
<li><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part4/#apfs-performance">Performance</a></li>
<li><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/#apfs-data">Data Integrity</a></li>
<li><a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part6/#apfs-conclusion">Conclusion</a></li>
</ul>
<h2>Basics</h2>
<p>APFS, the Apple File System, was itself started in 2014 with Dominic as its lead engineer. It’s a <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/#comment-55954">stand-alone, from-scratch implementation</a> (an earlier version of this post noted a dependency on <a href="https://en.wikipedia.org/wiki/Core_Storage">Core Storage</a>, but Dominic set me straight). I asked him about looking for inspiration in other modern file systems such as <a href="https://www.dragonflybsd.org/hammer/">BSD’s HAMMER</a>, <a href="https://btrfs.wiki.kernel.org/index.php/Main_Page">Linux’s btrfs</a>, or <a href="http://open-zfs.org/wiki/Main_Page">OpenZFS</a> (Solaris, illumos, <a href="https://www.freebsd.org/doc/handbook/zfs.html">FreeBSD</a>, <a href="https://openzfsonosx.org/">Mac OS X</a>, <a href="https://insights.ubuntu.com/2016/02/16/zfs-is-the-fs-for-containers-in-ubuntu-16-04/">Ubuntu Linux</a>, etc.), all of which have features similar to what APFS intends to deliver. (And note that Apple built a fairly complete port of ZFS, though Dominic was not apparently part of the group advocating for it.) Dominic explained that while, as a self-described file system guy (he built the file system in BeOS, unfairly relegated to obscurity when Apple opted to purchase NeXTSTEP instead), he was aware of them, but didn’t delve too deeply for fear, he said, of tainting himself.</p>
<p>Dominic praised the APFS testing team as being exemplary. This is absolutely critical. A common adage is that it takes a decade to mature a file system. And my experience with ZFS more or less confirms this. Apple will be delivering APFS broadly with 3-4 years of development so will need to accelerate quickly to maturity.<br />
<a></a></p>
<h2>Paying Down Debt</h2>
<p>HFS was introduced in 1985 when the Mac 512K (of memory! Holy smokes!) was Apple’s flagship. HFS+, a significant iteration, shipped in 1998 on the G3 PowerMacs with 4GB hard drives. Since then storage capacities have increased by factors of 1,000,000 and 1,000 respectively. HFS+ has been pulled in a bunch of competing directions with different forks for different devices (e.g. the iOS team created their own HFS variant, working so covertly that not even the Mac OS team knew) and different features (e.g. journaling, case insensitive). It’s old; it’s a mess; and, critically, it’s missing a bunch of features that are really considered the basic cost of doing business for most operating systems. <a href="https://en.wikipedia.org/wiki/HFS_Plus#Limitations">Wikipedia lists</a> nanosecond timestamps, checksums, snapshots, and sparse file support among those missing features. Add to that the obvious gap of large device support and you’ve got a big chunk of the APFS feature list.</p>
<p>APFS first and foremost pays down the unsustainable technical debt that Apple has been carrying in HFS+. (In 2001 ZFS grew from a similar need where UFS had been evolved since 1977.) It unifies the multifarious forks. It introduces the expected features. In general it first brings the derelict building up to code.</p>
<p>Compression is an obvious gap in the APFS feature list that is common in many file systems. It’s conceptually quite easy, I told the development team (we had it in ZFS from the outset), so why not include it? To appeal to Dominic’s BeOS nostalgia I even recalled my job interview with Be in 2000 when they talked about how compression actually improved overall performance since data I/O is far more expensive than computation (obvious now, but novel then). The Apple folks agreed, and—in typical Apple fashion—neither confirmed nor denied while strongly implying that it’s definitely a feature we can expect in APFS. I’ll be surprised if compression isn’t included in its public launch.</p>
<p><em>Next in this series: <a href="http://dtrace.org/blogs/ahl/2016/06/19/apfs-part2/">Encryption, Snapshots, and Backup</a></em></p>
 	
 	
 	
	
 </div>||||I||||
 Notice: register_sidebar_widget is deprecated since version 2.8! Use wp_register_sidebar_widget() instead. in /home/knmngmprl21d/public_html/blogs/wp-includes/functions.php on line 3467
 ﻿

 Thoughts on software, computing and technology

 * Home
 * About

 Adam Leventhal's blog

 From Prometheus to Sisyphus

 When Apple announced their new file system, APFS, in June, I hustled to be in the front row of the WWDC presentation, questions with the presenters, and then the open Q&A session. I took a week to write up my notes which turned into as 12 page behemoth of a blog post — longer than my college thesis. Despite reassurances from the tweeps , I was sure that the blog post was an order of magnitude longer than the modern attention span. I was wrong; so wrong that Ars Technica wanted to republish the blog post. Never underestimate the interest in all things Apple .

 In that piece I left one big thread dangling. Apple shipped APFS as a technology preview, but they left out access to one of the biggest new features: snapshots. Digging around I noticed that there was a curiously named new system call, “fs_snapshot”, but explicitly didn’t investigate: the post was already too long (I thought), I had spent enough time on it, and someone else (surely! surely?) would want to pull on that thread.

 Slow News Day

 Every so often I’d poke around for APFS news, but there was very little new. Last month folks discovered that APFS was coming to iOS sooner rather than later. But there wasn’t anything new to play with or any revelations on how APFS would work.

 I would search for “APFS snapshots”, “fs_snapshot”, anything I could think of to see if anyone had figured out how to make snapshots work on APFS. Nothing.

 A few weekends ago, I decided to yank on that thread myself.

 Prometheus

 I started from the system call, wandered through Apple’s open source kernel, leaned heavily on DTrace, and eventually figured it out. Apple had shipped snapshots in APFS, they just hadn’t made it easy to get there. The folks at Ars were excited for a follow-up, and my investigation turned into this: “Testing out snapshots in Apple’s next-generation APFS file system.”

 Snapshots were there; the APIs were laid bare; I was going to bring fire to all the Mac fans; John Siracusa and Andy Ihnatko would carry me on their shoulders down the streets of the Internet.

 Sisyphus

 On the eve that this new piece was about to run I was nervously scrolling through Twitter as I took the bus home from work. Now that I had invested the time to research and write-up APFS snapshots I didn’t want someone else beating me to the punch.

 Then I found this and my heart dropped:

 Slides from the “Storing our digital lives: Mac filesystems from MFS to APFS” session at MacTech Conference 2016: https://t.co/uJJuqLL8n9

 — Rich Trouton (@rtrouton) November 17, 2016

 Skim past the craziness of MFS and the hairball of HFS, and start digging through the APFS section. Slide 49, “APFS Snapshots” and there it is “apfs_snapshot” — not a tool that anyone laboriously reverse engineered, deciphering system calls and semi-published APIs — a tool shipped from Apple and included in macOS by default. F — .

 Apple had secreted this utility away (along with some others) in /System/Library/Filesystems/apfs.fs/Contents/Resources/

 What To Do?

 The article that was initially about a glorious act of discovery had become an article about the reinvention of the wheel. Conversely, vanishingly few people would recognize this as rediscovery since the apfs_snapshot tool was so obscure (9 hits on Google!).

 We toned down the already modest chest-thumping and published the article this morning to a pretty nice response so far. I might have happier as an FAKE NEWS Prometheus, blissfully unaware of the pre-existence of fire, but I would have been mortified when the inevitable commenter, one of the few who had used apfs_snapshot, crushed me with my own boulder.

 Posted on February 12, 2017 at 5:23 pm by ahl · Permalink · One Comment
 In: DTrace

 DTrace at Home

 I had been procrastinating making the family holiday card. It was a combination of having a lot on my plate and dreading the formulation of our annual note recapping the year; there were some great moments, but I’m glad I don’t have to do 2016 again. It was just before midnight and either I’d make the card that night or leave an empty space on our friends’ refrigerators. Adobe Illustrator had other ideas:

 I’m not the first person to hit this. The problem seems to have existed since CS6 was released in 2016. None of the solutions was working for me, and — inspired by Sara Mauskopf’s excellent post — I was rapidly running out of the time bounds for the project. Enough; I’d just DTrace it.

 A colleague scoffed the other day, “I mean, how often do you actually use DTrace?” In his mind DTrace was for big systems, critical system, when dollars and lives were at stake. My reply: I use DTrace every day. I can’t imagine developing software without DTrace, and I use it when my laptop (not infrequently) does something inexplicable (I’m forever grateful to the Apple team that ported it to Mac OS X).

 First I wanted to make sure I had the name of the Illustrator process right:

 $ sudo dtrace -n ‘syscall:::entry{ @[execname] = count(); }’
 dtrace: description ‘syscall:::entry’ matched 500 probes
 ^C
 pboard 1
 watchdogd 2
 awdd 3
 ...
 com.apple.WebKit 7065
 Google Chrome He 7128
 Google Chrome 8099
 Adobe Illustrato 36674

 Glad I checked: “Adobe Illustrato”. Now we can be pretty sure that Illustrator is failing on setrlimit(2) and blowing up as result. Let’s confirm that it is in fact returning -1:

 $ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato"/{ printf("%d %d", arg1, errno); }'
 dtrace: description 'syscall::setrlimit:return' matched 1 probe
 CPU ID FUNCTION:NAME
 0 532 setrlimit:return -1 1

 There it is. And setrlimit(2) is failing with errno 1 which is EPERM (value too high for non-root user). I already tuned up the files limit pretty high. Let’s confirm that it is in fact setting the files limit and check the value to which it’s being set. To write this script I looked at the documentation for setrlimit(2) (hooray for man pages!) to determine that the position of the resource parameter (arg0) and the type of the value parameter (struct rlimit). I needed the DTrace copyin() subroutine to grab the structure from the process’s address space:

 $ sudo dtrace -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->r = *(struct rlimit *)copyin(arg1, sizeof (struct rlimit)); printf("%x %x %x", arg0, this->r.rlim_cur, this->r.rlim_max); }'
 dtrace: description 'syscall::setrlimit:entry' matched 1 probe
 CPU ID FUNCTION:NAME
 0 531 setrlimit:entry 1008 2800 7fffffffffffffff

 Looking through /usr/include/sys/resource.h we can see that 1008 corresponds to the number of files (RLIMIT_NOFILE | _RLIMIT_POSIX_FLAG). Illustrator is trying to set that value to 0x7fffffffffffffff or 2⁶³-1. Apparently too big; I filed any latent curiosity for another day.

 The quickest solution was to use DTrace again to whack a smaller number into that struct rlimit. Easy:

 $ sudo dtrace -w -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this->i = (rlim_t *)alloca(sizeof (rlim_t)); *this->i = 10000; copyout(this->i, arg1 + sizeof (rlim_t), sizeof (rlim_t)); }'
 dtrace: description 'syscall::setrlimit:entry' matched 1 probe
 dtrace: could not enable tracing: Permission denied

 Oh right. Thank you SIP. This isa new laptop (at least a new motherboard due to some bizarre issue) which probably contributed to Illustrator not working when once it did. Because it’s new I haven’t yet disabled the part of SIP that prevents you from using DTrace on the kernel or in destructive mode (e.g. copyout()). It’s easy enough to disable, but I’m reboot-phobic — I hate having to restart my terminals — so I went to plan B: lldb.

 First I used DTrace to find the code that was calling setrlimit(2): using some knowledge of the x86 ISA/ABI:

 $ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato" && arg1 == -1/{ printf("%x", *(uintptr_t *)copyin(uregs[R_RSP], sizeof (uintptr_t)) - 5) }'
 dtrace: description 'syscall::setrlimit:return' matched 1 probe
 CPU ID FUNCTION:NAME
 0 532 setrlimit:return 1006e5b72
 0 532 setrlimit:return 1006e5b72

 I ran it a few times to confirm the address of the call instruction and to make sure the location wasn’t being randomized. If I wasn’t in a rush I might have patched the binary, but Apple’s Mach-O Object format always confuses me. Instead I used lldb to replace the call with a store of 0 to %eax (to evince a successful return value) and some nops as padding (hex values I remember due to personal deficiencies):

 (lldb) break set -n _init
 Breakpoint 1: 47 locations.
 (lldb) run
 ...
 (lldb) di -s 0x1006e5b72 -c 1
 0x1006e5b72: callq 0x1011628e0 ; symbol stub for: setrlimit
 (lldb) memory write 0x1006e5b72 0x31 0xc0 0x90 0x90 0x90
 (lldb) di -s 0x1006e5b72 -c 4
 0x1006e5b72: xorl %eax, %eax
 0x1006e5b74: nop
 0x1006e5b75: nop
 0x1006e5b76: nop

 Next I just process detach and got on with making that holiday card…

 DTrace Every Day

 DTrace was designed for solving hard problems on critical systems, but the need to understand how systems behave exists in development and on consumer systems. Just because you didn’t write a program doesn’t mean you can’t fix it.

 Posted on December 18, 2016 at 4:32 am by ahl · Permalink · 2 Comments
 In: DTrace

 A Filesystem on Noms

 Since Noms dropped last week the dev community has seemed into it. “Git for data” — it simultaneously evokes something very familiar and yet unconstrained. Something that hasn’t been well-noted is how much care the team has taken to make Noms fun to build with, and it is.


 The Adobe Flash Player is required for video playback.
 Get the latest Flash Player or Watch this video on YouTube .

 Noms is a content-addressable, decentralized, append-only database. It borrows concepts from a variety of interesting data systems. Obviously databases are represented: Noms is a persistent, transactional data repository. You can also see the fundamentals of git and other decentralized source code management tools. Noms builds up a chain of commits; those chains can be extended, forked, and shared, while historical data are preserved. Noms shares much in common with modern filesystems such as ZFS, btrfs, or Apple’s forthcoming APFS. Like those filesystems, Noms uses copy-on-write, never modifying data in situ; it forms a self-validating hierarchy of data; and it (intrinsically) supports snapshots and efficient movement of data between snapshots.

 After learning a bit about Noms I thought it would be interesting to use it as the foundation for a filesystem. I wanted to learn about Noms, and contribute a different sort of example that might push the project in new and valuable ways. The Noms founders, Aaron and Raf, were fired up so I dove in…

 What’s Modern?

 When people talk about a filesystem being “modern” there’s a list of features that they often have in mind. Let’s look at how the Noms database stacks up:

 Snapshots

 A filesystem snapshot preserves the state of the filesystem for some future use — typically data recovery or fast cloning. Since Noms is append-only, every version is preserved. Snapshots are, therefore, a natural side effect. You can make a Noms “snapshot” — any commit in a dataset’s history — writeable by syncing it to a new dataset. Easy.

 Dedup

 The essence of dedup is that unique data should be stored exactly once. If you duplicate a file, a folder, or an entire filesystem the storage consumption should be close to zero. Noms is content addressable, unique data is only ever stored once. Every Noms dataset intrinsically removes duplicated data.

 Consistency

 A feature of a filesystem — arguably the feature of a filesystem — is that it shouldn’t ever lose or corrupt your data. One common technique to ensure consistency is to write new data to a new location rather than overwriting old data — so called copy-on-write (COW). Noms is append-only, it doesn’t throw out (or overwrite) old data; copying modified is required and explicit. Noms also recursively checksums all data — a feature of ZFS and btrfs, notably absent from APFS.

 Backup

 The ability to backup your data from a filesystem is almost as important as keeping it intact in the first place. ZFS, for example, lets you efficiently serialize and send the latest changes between systems. When pulling or pushing changes git also efficiently serializes just the changed data. Noms does something similar with its structured data. Data differences are efficiently computed to optimize for minimal data transfer.

 Noms has all the core components of a modern filesystem. My goal was to write the translation layer to expose filesystem semantics on top of the Noms interfaces.

 Designing a Schema

 Initially, Badly

 It’s in the name: Noms eats all the data. Feed it whatever data you like, and let Noms infer a schema as you go. For a filesystem though I wanted to define a fixed structure. I started with a schema modeled on a simplified ZFS. Filesystems keep track of files and directories with a structure called an “inode” each of which has a unique integer identifier, the “inode number”. ZFS keeps track of files and directories with DMU objects named by their integer ID. The schema would use a Map<number, Inode> to serve the same function (spoiler: read on and don’t copy this!):

 
 struct Filesystem {
 inodes: Map<Number, struct Inode {
 attr: struct Attr { /* e.g. permissions, modification time, etc. */ }
 contents: Union {
 struct File { data: Ref /* Noms pile of bytes */ } |
 struct Directory { contents: Map<String, Number> }
 }
 }>
 rootInode: Number
 maxInode: Number
 }
 

 Nice and simple. Files are just Noms Blobs represented by a sequence of bytes. Directories are a Map of strings (the name of the directory entry) to the inode number; the inode number can be used to find the actual content by looking in the top-level map.

 Schema philosophy

 This made sense for a filesystem. Did it make sense for Noms? I wasn’t trying to put the APFS team out of work, rather I was creating a portal from the shell or Finder into Noms. To evaluate the schema, I had the benefit of direct access to the Noms team (and so can all developers at http://slack.noms.io/). I learned two guiding principles for data in Noms:

 Noms data should be self-validating. As much as possible the application should rely on noms to ensure consistency. My schema failed this test because the relationship between inode numbers contained in directories and the entires of the inodes map was something my code alone could maintain and validate.

 Noms data should be deterministic. A given collection of data should have a single representation; the Noms structures should be path and history independent. Two apparently identical datasets should be identical in the eyes of Noms to support efficient storage and transfer, and identical data should produce an identical hash at the root of the dataset. Here, again, my schema fell short because the inode number assigned to a given file or directory depended on how other objects were created. Two different users with two identical filesystems wouldn’t be able to simply sync the way they would with two identical git branches.

 A Better Schema

 My first try made for a fine filesystem, just not a Noms filesystem. With a better understanding of the principles, and with help from the Noms team, I built this schema:

 
 struct Filesystem {
 root: struct Inode {
 attr: struct Attr { /* e.g. permissions, modification time, etc. */ }
 contents: Union {
 struct File { data: Ref<Blob> /* Noms pile of bytes */ } |
 struct Directory: { contents: Map<string, Cycle<1>> }
 }
 }
 }
 

 Obviously simpler; the thing that bears explanation is the use of so-called “Cycle” types. A Cycle is a means of expressing a recursive relationship within Noms types. The integer parameter specifies the ancestor struct to which the cycle refers. Consider a linked list type:

 struct LinkedList {
 data: Blob
 next: Cycle<0>
 }

 The “next” field refers to immediately containing struct, LinkedList. In our filesystem schema, a Directory’s contents are represented by a map of strings (directory entry names) to Inodes, Cycle<1> referring to the struct “above” or “containing” the Directory type. (Read on for a visualization of this.)

 Writing It

 To build the filesystem I picked a FUSE binding for Go, dug into the Noms APIs, and wrestled my way through some Go heartache.

 Working with Noms requires a slightly different mindset than other data stores. Recall in particular that Noms data is immutable. Adding a new entry into a Map creates a new Map. Setting a member of a Struct creates a new Struct. Changing nested structures such as the one defined by our schema requires unzipping it, and then zipping it back together. Here’s a Go snippet that demonstrates that methodology for creating a new directory:

 Demo

 Showing it off has all the normal glory of a systems demo! Check out the documentation for requirements.

 Create and mount a filesystem from a new local Noms dataset:

 $ go build
 $ mkdir /var/tmp/mnt
 $ go run nomsfs.go /var/tmp/nomsfs::fs /var/tmp/mnt
 running...

 You can open the folder and drop data into it.

 Your database fell into my filesystem!

 Now let’s take a look at the underlying Noms dataset:

 $ noms show http://demo.noms.io/ahl_blog::fs
 struct Commit {
 meta: struct {},
 parents: Set<Ref<Cycle<0>>>,
 value: struct Filesystem {
 root: struct Inode {
 attr: struct Attr {
 ctime: Number,
 gid: Number,
 mode: Number,
 mtime: Number,
 uid: Number,
 xattr: Map<String, Blob>,
 },
 contents: struct Directory {
 entries: Map<String, Cycle<1>>,
 } | struct Symlink {
 targetPath: String,
 } | struct File {
 data: Ref<Blob>,
 },
 },
 },
 }({
 meta: {},
 parents: {
 5v82rie0be68915n1q7pmcdi54i9tmgs,
 },
 value: Filesystem {
 root: Inode {
 attr: Attr {
 ctime: 1.4705227450393803e+09,
 gid: 502,
 mode: 511,
 mtime: 1.4705227450393803e+09,
 uid: 110853,
 xattr: {},
 },
 contents: Directory {
 entries: {
 "usenix_winter91_faulkner.pdf": Inode {
 attr: Attr {
 ctime: 1.4705228859273868e+09,
 gid: 502,
 mode: 420,
 mtime: 1.468425783e+09,
 uid: 110853,
 xattr: {
 "com.apple.FinderInfo": 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 // 32 B
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00,
 "com.apple.quarantine": 30 30 30 31 3b 35 37 38 36 36 36 33 37 3b 53 61 // 21 B
 66 61 72 69 3b,
 },
 },
 contents: File {
 data: dmc45152ie46mn3ls92vvhnm41ianehn,
 },
 },
 },
 },
 },
 },
 })

 You can see the type at the top and then the actual filesystem contents. Let’s look at more complicated example where I’ve taken part of the Noms source tree and copied it to nomsfs. You can navigate around its structure courtesy of the Splore utility (take particular note of nested directories that show the recursive data definition described above):

 Embedded ‘Splore! http://splore.noms.io/?db=https://demo.noms.io/ahl_blog&hash=2nhi5utm4s38hka22vt9ilv5i3l8r2ol

 You can see the all of the various states that the filesystem has been through — each state change — using noms log http://demo.noms.io/ahl_blog::fsnoms. You can sync it to your local computer with noms sync http://demo.noms.io/ahl_blog::fsnoms /var/tmp/fs or checkout some previous state from the log (just like a filesystem snapshot). Diff two states from the log or make your own changes and diff it with the original using noms diff.

 Nom Nom Nom

 It took less than 1000 lines of Go code to make Noms appear as a Window in the Finder, eating data as quickly as I could drag and drop (try it!). Imagine what Noms might look like behind other known data interfaces; it could bring git semantics to existing islands of data. Noms could form the basis of a new type of data lake — maybe one that’s simple and powerful enough to bring real results. Beyond the marquee features, Noms is fun to build with. I’m already working on my next Noms application.

 Posted on August 9, 2016 at 3:54 pm by ahl · Permalink · 5 Comments
 In: Software · Tagged with: APFS, btrfs, fuse, go, noms, ZFS

 I Love Go; I Hate Go

 I liked Go right away. It was close enough to C and Java to be instantly familiar, the examples and tutorials were straightforward, and I was quickly writing real code. I’ve wanted to learn Go since its popularity was surging few years ago. In no danger of being judged an early adopter, I happily found a great project that—as it happened—had to be in Go (more in a future post).

 I Love Go

 My first priority was not looking stupid. The folks I’d be doing this for are actual Go developers; I wanted my code to fit in without imposing with too many questions. They had no style guide. Knowing that my 80-column width sensibilities expose an unattractive nostalgia, I went looking for a max line length. There wasn’t one. Then I discovered gofmt, the simple tool that Go employs to liberate developers from the tyranny of stylistic choice. It takes Go code and spits it back out in the One True Style. I love it. I was raised in an engineering culture with an exacting style guide, but any style guide has gaps. Factions form; style-originalists face off against those who view (incorrectly!) the guide as a living document. I updated my decades-old .vimrc to gofmt on save. These Go tyrants were feeling like my kind of tyrant.

 One of the things that turned me off of C++ (98, 11, and 14) is its increasing amount of magic. Go is comparatively straightforward. When I reach for something I find that it’s where I expect it to be. Libraries and examples aren’t mysterious. Error messages are non-mysterious (other than quickly resolved confusion about methods with “pointer receivers”). Contrast this with Rust whose errors read like mind-bindingly inscrutable tax forms.

 Go’s containment-based inheritance is easy enough to reason about. Its interfaces are similarly no-nonsense and pragmatic. You don’t have to define a structure as implementing an interface. You can use an interface to describe anything that implements it. Simple. This is particularly useful for testing. In a messy environment with components beyond your control you can apply interfaces to other people’s code, and then mock it out.

 The toolchain is, again, simple to use—the benefit of starting from scratch—and makes for fast compilation, quick testing, and easy integration with a good-sized ecosystem. I stopped worrying about dependencies, rebuilding, etc. knowing that go run would find errors wherever I had introduced them and do so remarkably quickly.

 I Hate Go

 Go is opinionated. Most successful products have that strong sense of what they are and what they aren’t; Go draws as sharp a line as any language. I was seduced by its rightheadedness around style, but with anything or anyone that opinionated you’ll find some of those opinions weird and others simply off-putting.

 Reading the official documentation, I found myself in the middle of a section prefixed with the phrase “if GOOS is set to plan9”. Wow. I’m a few standard deviations from the norm in terms of being an OS nerd, but I’ve never even seen Plan 9. I knew that the Plan 9 folks got the band back together to create Go; it’s great that their pop audiences don’t dissuade them from playing off their old B-sides. Quirky, but there’s nothing wrong with that.

 I wanted to check an invariant; how does Go do assertions? Fuck you, you’re a bad programmer. That’s how. The Go authors feel so strongly about asserts being typically misused that they refuse to provide them. So folks use one (or more) of some workable libraries.

 I created an infinite recursion, overflowing the stack. Go produces the first 100 stack frames and that’s it. Maybe you can change that, but I couldn’t figure out how. (“go stackoverflow” is about the most useless thing you can search for; chapeau, Go and Stackoverflow respectively.) I could be convinced that I only want 100 stack frames, but not the last 100, not the same 4 over and over again. I ended up limiting the stack size with runtime.debug.SetMaxStack(), Goldilocks-ing it between too big to catch the relevant frames and too small to allow for normal operation.

 I tried using other tools (ahem, DTrace) to print the stack, but, of course, the Go compiler omits frame pointers rendering the stacks unobservable to debuggers. Ditto arguments due to ABI-non-compliant calling conventions, but that’s an aside. The environment variable GOEXPERIMENT=framepointer is supposed to compile with frame pointers, but it was a challenge to rebuild the world. All paths seemed to lead me to my former-colleague’s scathing synopsis: Golang is Trash.

 As fun as it is to write code in Go, debugging in Go is no fun at all. I may well just be ignorant of the right tooling. But there sure isn’t a debugger with the simple charm of go build for compilation, go test for testing, or go run for execution.

 Immaturity and Promise

 Have you ever been in a relationship where minor disagreements immediately escalated to “maybe we should break up?” The Go documentation even seems ready to push you to some other language at the slightest affront. Could I have asserts? Sure, if you’re a bad programmer. Perhaps ABI compliance has its merits? I’m sure you could find that in some other language. Could you give me the absolute value of this int? Is something wrong with your ‘less than’ key?

 I’m sure time will, as it tends to, bring pragmatism. I appreciate that Go does have strong opinions (it’s just hard to remember that when I disagree with them). Weak opinions are what turn languages into unreadable mishmashes of overlapping mechanism. My favorite example of this is Perl. My first real programming job was in Perl. I was the most avid reader teenage of the Perl llama and camel books. Ask me about my chat with Larry Wall, Perl’s creator, if you see a beer in my hand. In an interview, Larry said, “In Perl 6, we decided it would be better to fix the language than fix the user”. Contrast this with Go’s statement on assertions:

 Go doesn’t provide assertions. They are undeniably convenient, but our experience has been that programmers use them as a crutch to avoid thinking about proper error handling and reporting.

 Perl wants to be whatever the user wants to be. The consumate pleaser. Go sees the user as flawed and the strictures of the language as the cure. It’s an authoritarian, steadfast in its ideals, yet too sensitive to find compromise (sound like anyone we all know?).

 Despite my frustrations I really enjoy writing code in Go. It’s clean and supported by a great community and ecosystem. I’m particularly heartened that Go 1.7 will compile with frame pointers by default. Diagnosing certain types of bugs is still a pain in the neck, but I’m sure it will improve and I’ll see where I can pitch in.

 Posted on August 2, 2016 at 3:19 am by ahl · Permalink · 23 Comments
 In: Software

 APFS in Detail: Conclusions

 This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the first post for the table of contents.

 Summing Up

 I’m not sure Apple absolutely had to replace HFS+, but likely they had passed an inflection point where continuing to maintain and evolve the 30+ year old software was more expensive than building something new. APFS is a product born of that assessment.

 Based on what Apple has shown I’d surmise that its core design goals were:

 * satisfying all consumers (laptop, phone, watch, etc.)
 * encryption as a first-class citizen
 * snapshots for modernized backup.

 Those are great goals that will benefit all Apple users, and based on the WWDC demos APFS seems to be on track (though the macOS Sierra beta isn’t quite as far along).

 In the process of implementing a new file system the APFS team has added some expected features. HFS was built when 400KB floppies ruled the Earth (recognized now only as the ubiquitous and anachronistic save icon). Any file system started in 2014 should of course consider huge devices, and SSDs–check and check. Copy-on-write (COW) snapshots are the norm; making the Duplicate command in the Finder faster wasn’t much of a detour. The use case is unclear, it’s a classic garbage can theory solution, a solution in search of a problem, but it doesn’t hurt and it makes for a fun demo. The beach ball of doom earned its nickname; APFS was naturally built to avoid it.

 There are some seemingly absent or ancillary design goals: performance, openness, and data integrity. Squeezing the most IOPS or throughput out of a device probably isn’t critical on watchOS, and it’s relevant only to a small percentage of macOS users. It will be interesting to see how APFS performs once it ships (measuring any earlier would only misinform the public and insult the APFS team). The APFS development docs have a bullet on open source: “An open source implementation is not available at this time.” I don’t expect APFS to be open source at this time or any other, but prove me wrong, Apple. If APFS becomes world-class I’d love to see it in Linux and FreeBSD–maybe Microsoft would even jettison their ReFS experiment. My experience with OpenZFS has shown that open source accelerates that path to excellence. It’s a shame that APFS lacks checksums for user data and doesn’t provide for data redundancy. Data integrity should be job one for a file system, and I believe that that’s true for a watch or phone as much as it is for a server.

 At stability, APFS will be an improvement, for Apple users of all kinds, on every device. There are some clear wins and some missed opportunities. Now that APFS has been shared with the world the development team is probably listening. While Apple is clearly years past the decision to build from scratch rather than adopting existing modern technology, there’s time to raise the priority of data integrity and openness. I’m impressed by Apple’s goal of using APFS by default within 18 months. Regardless of how it goes, it will be an exciting transition.

 Posted on June 19, 2016 at 7:37 pm by ahl · Permalink · 37 Comments
 In: Software · Tagged with: APFS

 APFS in Detail: Data Integrity

 This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the first post for the table of contents.

 Data Integrity

 Arguably the most important job of a file system is preserving data integrity. Here’s my data, don’t lose it, don’t change it. If file systems could be trusted absolutely then the “only” reason for backup would be the idiot operators (i.e. you and me). There are a few mechanisms that file systems employ to keep data safe.

 Redundancy

 APFS makes no claims with regard to data redundancy. As Apple’s Eric Tamura noted at WWDC, most Apple devices have a single storage device (i.e. one logical SSD) making RAID, for example, moot. Instead redundancy comes from lower layers such as Apple RAID (apparently a thing), hardware RAID controllers, SANs, or even the “single” storage devices themselves..

 As an aside note that SSDs in most Apple products where APFS will run include multiple more-or-less independent NAND chips. High-end SSDs do implement data redundancy within the device, but it comes at the price of reduced capacity and performance. As noted above, the “flash-optimization” of APFS doesn’t actually extend much below the surface of the standard block device interface, but the raw materials for innovation are there.

 Also, APFS removes the most common way of a user achieving local data redundancy: copying files. A copied file in APFS actually creates a lightweight clone with no duplicated data. Corruption of the underlying device would mean that both “copies” were damaged whereas with full copies localized data corruption would affect just one.

 Crash Consistency

 Computer systems can fail at any time—crashes, bugs, power outages, etc.—so file systems need to anticipate and recover from these scenarios. The old old old school method is to plod along and then have a special utility to check and repair the file system during boot (fsck, short for file system check). More modern systems labor to achieve an always consistent format, or only narrow windows of inconsistency, obviating the need for the full, expensive fsck. ZFS, for example, builds up new state on disk and then atomically transitions from the previous state to the new one with a single atomic operation.

 Overwriting data creates the most obvious opening for inconsistency. If the file system needs to overwrite several regions there is a window where some regions represent the new state and some represent the former state. Copy-on-write (COW) is a method to avoid this by always allocating new regions and then releasing old ones for reuse rather than modifying data in-place. APFS claims to implement a “novel copy-on-write metadata scheme”; APFS lead developer Dominic Giampaolo emphasized the novelty of this approach without delving into the details. In conversation later, he made it clear that APFS does not employ the ZFS mechanism of copying all metadata above changed user data which allows for a single, atomic update of the file system structure.

 It’s surprising to see that APFS includes fsck_apfs—even after asking Dominic I’m not sure why it would be necessary. For comparison I don’t believe there’s been an instance where fsck for ZFS would have found a problem that the file system itself didn’t already know how to detect. But Dominic was just as confused about why ZFS would forego fsck, so perhaps it’s just a matter of opinion.

 Checksums

 Notably absent from the APFS intro talk was any mention of checksums. A checksum is a digest or summary of data used to detect (and correct) data errors. The story here is surprisingly nuanced. APFS checksums its own metadata but not user data. The justification for checksumming metadata is strong: there’s relatively not much of it (so the checksums don’t consume much storage) and losing metadata can cast a potentially huge shadow of data loss. If, for example, metadata for a top level directory is corrupted then potentially all data on the disk could be rendered inaccessible. ZFS duplicates metadata (and triple duplicates top-level metadata) for exactly this reason.

 Explicitly not checksumming user data is a little more interesting. The APFS engineers I talked to cited strong ECC protection within Apple storage devices. Both flash SSDs and magnetic media HDDs use redundant data to detect and correct errors. The engineers contend that Apple devices basically don’t return bogus data. NAND uses extra data, e.g. 128 bytes per 4KB page, so that errors can be corrected and detected. (For reference, ZFS uses a fixed size 32 byte checksum for blocks ranging from 512 bytes to megabytes. That’s small by comparison, but bear in mind that the SSD’s ECC is required for the expected analog variances within the media.) The devices have a bit error rate that’s tiny enough to expect no errors over the device’s lifetime. In addition, there are other sources of device errors where a file system’s redundant check could be invaluable. SSDs have a multitude of components, and in volume consumer products they rarely contain end-to-end ECC protection leaving the possibility of data being corrupted in transit. Further, their complex firmware can (does) contain bugs that can result in data loss.

 The Apple folks were quite interested in my experience with regard to bit rot (aging data silently losing integrity) and other device errors. I’ve seen many instances where devices raised no error but ZFS (correctly) detected corrupted data. Apple has some of the most stringent device qualification tests for its vendors; I trust that they really do procure the best components. Apple engineers I spoke with claimed that bit rot was not a problem for users of their devices, but if your software can’t detect errors then you have no idea how your devices really perform in the field. ZFS has found data corruption on multi-million dollar storage arrays; I would be surprised if it didn’t find errors coming from TLC (i.e. the cheapest) NAND chips in some of Apple’s devices. Recall the (fairly) recent brouhaha regarding storage problems in the high capacity iPhone 6. At least some of Apple’s devices have been imperfect.

 As someone who has data he cares about on a Mac, who has seen data lost from HFS, and who knows that even expensive, enterprise-grade equipment can lose data, I would gladly sacrifice 16 bytes per 4KB–less than 1% of my device’s size.

 Scrub

 As data ages you might occasionally want to check for bit rot. Likely fsck_apfs can accomplish this; as noted though there’s no data redundancy and no checksums for user data, so scrub would only help to find problems and likely wouldn’t help to correct them. And if it makes it any easier for Apple to reverse course, let’s say it’s for the el cheap-o drive I bought from Fry’s not for the gold-plated device I got from Apple.

 Next in this series: Conclusions

 Posted on June 19, 2016 at 7:37 pm by ahl · Permalink · 41 Comments
 In: Software · Tagged with: APFS, checksums, RAID

 APFS in Detail: Performance

 This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the first post for the table of contents.


 Performance

 APFS claims to be optimized for flash. Flash memory (NAND) is the stuff in your speedy SSD. Apple changed the computing industry when it put flash into the iPod and iPhone, volumes for which fundamentally changed the economics of flash. This consumer change impacted the enterprise (as it often does), giving rise to hybrid and all-flash arrays. Ten years ago flash cost as much as DRAM; now it’s challenging the economics of hard disks.

 SSDs mimic the block interface of conventional hard drives, but the underlying technology is completely different. In particular while magnetic media can read or write sectors arbitrarily, flash erases large chunks (blocks) and reads and writes smaller chunks (pages). The management is done by what’s called the flash translation layer (FTL), software that makes blocks and pages appear more like a hard drive. An FTL is very similar to a file system, creating a virtual mapping (a translation) between block addresses and locations within the media. Apple controls the full stack including the SSD, FTL, and file system; they could have built something differentiated, optimizing this components to work together. What APFS does, however, is simply write in patterns known to be more easily handled by NAND. It’s a file system with flash-aware characteristics rather than one written explicitly for the native flash interfaces, more or less what you’d expect in 2016.

 Also on the topic of flash, APFS includes TRIM support. TRIM is a command in the ATA protocol that allows a file system to indicate to an SSD (specifically, its FTL) that some space has been freed. SSDs require significant free space and perform better when there’s more of it; they include more physical space than they advertise. For example, my 1TB SSD includes 1TB (240 = 10244) bytes of flash but only reports 931GB of available space, sneakily matching the storage industry’s self-serving definition of 1TB (10004 = 1 trillion bytes). With more free space, FTLs can trade off space efficiency for performance and longevity. TRIM has become expected of file systems; it’s unsurprising that APFS supports it. The problem with TRIM though is that it’s only useful when there’s free space: it’s something of a benchmark special. Once your disk is mostly full (as mine are in my laptop and phone basically at all times) TRIM doesn’t do anything for you. I doubt that TRIM will bring any discernible benefit for APFS users beyond the placebo effect of feature parity.

 APFS also focuses on latency; Apple’s number one goal is to avoid the beachball of doom. APFS addresses this with I/O QoS (quality of service) to prioritize accesses that are immediately visible to the user over background activity that doesn’t have the same time-constraints. This is inarguably a benefit to users and a sophisticated file system capability.

 Next in this series: Data Integrity

 Posted on June 19, 2016 at 7:36 pm by ahl · Permalink · 11 Comments
 In: Software · Tagged with: APFS, flash, FTL, NAND, Performance, SSD, TRIM

 APFS in Detail: Space Efficiency and Clones

 This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the first post for the table of contents.

 Space Efficiency

 A modern trend in file systems has been to store data more efficiently to effectively increase the size of your device. Common approaches include compression (which, as noted above, is very very likely coming) and deduplication. Dedup finds common blocks and avoids storing them multiply. This is potentially highly beneficial for file servers where many users or many virtual machines might have copies of the same file; it’s probably not useful for the single-user or few-user environments that Apple cares about. (Yes, they have server-ish offerings but their heart clearly isn’t into it.) It’s also furiously hard to do well as I learned painfully while supporting ZFS.

 Apple’s sort-of-unique contribution to space efficiency is constant time cloning of files and directories. As a quick aside, “files” in macOS are often really directories; it’s a convenient lie they tell to allow logically related collections of files to be treated as an indivisible unit. Right click an application and select “Show Package Contents” to see what I mean. Accordingly, I’m going to use the term “file” rather than “file or directory” in sympathy for the patient readers who have made it this far.

 With APFS, if you copy a file within the same file system (or possibly the same container; more on this later), no data is actually duplicated. Instead a constant amount of metadata is updated and the on-disk data is shared. Changes to either copy cause new space to be allocated (so-called “copy on write” or COW).

 I haven’t seen this offered in other file systems, and it clearly makes for a good demo, but it got me wondering about the use case (UPDATE: btrfs supports this and calls the feature “reflinks”–link by reference). Copying files between devices (e.g. to a USB stick for sharing) still takes time proportional to the amount of data copied of course. Why would I want to copy a file locally? The common case I could think of is the layman’s version control: “thesis”, “thesis-backup”, “thesis-old”, “thesis-saving because I’m making edits while drunk”.

 There are basically three categories of files:

 * Files that are fully overwritten each time; images, MS Office docs, videos, etc.
 * Files that are appended to, mostly log files
 * Files with a record-based structure, such as database files.

 For the average user, most files fall into that first category. So with APFS I can make a copy of my document and get the benefits of space sharing, but those benefits will be eradicated as soon as I save the new revision. Perhaps users of larger files have a greater need for this and have a better idea of how it might be used.

 Personally, my only use case is taking a file, say time-shifted Game of Thrones episodes falling into the “fair use” section of copyright law, and sticking it in Dropbox. Currently I need to choose to make a copy or permanently move the file to my Dropbox folder. Clones would let me do this more easily. But then so would hard links (a nearly ubiquitous file system feature that lets a file appear in multiple directories).

 Clones open the door for potential confusion. While copying a file may take up no space, so too deleting a file may free no space. Imagine trying to free space on your system, and needing to hunt down the last clone of a large file to actually get your space back.

 APFS engineers don’t seem to have many use cases in mind; at WWDC they asked for suggestions from the assembled developers (the best I’ve heard is for copied VMs; not exactly a mass-market problem). If the focus is generic revision control, I’m surprised that Apple didn’t shoot for a more elegant solution. One could imagine functionality with APFS that allows a user to enable per-file Time Machine, change tracking for any file. This would create a new type of file where each version is recorded transparently and automatically. You could navigate to previous versions, prune the history, or delete the whole pile of versions at once (with no stray clones to hunt down). In fact, Apple introduced something related 5 years ago, but I’ve literally never seen or heard of it until researching this post (show of hands if you’ve clicked “Browse All Versions…”). APFS could clean up its implementation, simplify its use, and bring generic support for all applications. None of this solves my Game of Thrones storage problem, but I’m not even sure it’s much of a problem…

 Side note: Finder copy creates space-efficient clones, but cp from the command line does not.

 Next in this series: Performance

 Posted on June 19, 2016 at 7:35 pm by ahl · Permalink · 24 Comments
 In: Software · Tagged with: APFS, snapshots

 APFS in Detail: Encryption, Snapshots, and Backup

 This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the first post for the table of contents.

 Encryption

 Encryption is clearly a core feature of APFS. This comes from diverse requirements from the various devices, for example multiple keys within file systems on the iPhone or per-user keys on laptops. I heard the term “innovative” quite a bit at WWDC, but here the term is aptly applied to APFS. It supports several different encryption choices for a file system:

 * Unencrypted
 * Single-key for metadata and user data
 * Multi-key with different choices for metadata, files, and even sections of a file (“extents”)

 Multi-key encryption is particularly relevant for portables where all data might be encrypted, but unlocking your phone provides access to an additional key and therefore additional data. Unfortunately this doesn’t seem to be working in the first beta of macOS Sierra (specifying fileEncryption when creating a new volume with diskutil results in a file system that reports “Is Encrypted” as “No”).

 Related to encryption, I noticed an undocumented feature while playing around with diskutil (which prompts you for interactive confirmation of the destructive power of APFS unless this is added to the command-line: -IHaveBeenWarnedThatAPFSIsPreReleaseAndThatIMayLoseData; I’m not making this up). APFS (apparently) supports constant time cryptographic file system erase, called “effaceable” in the diskutil output. This presumably builds a secret key that cannot be extracted from APFS and encrypts the file system with it. A secure erase then need only delete the key rather than needing to scramble and re-scramble the full disk to ensure total eradication. Various iOS docs refer to this capability requiring some specialized hardware; it will be interesting to see what the option means on macOS. Either way, let’s not mention this to the FBI or NSA, agreed?

 Snapshots and Backup

 APFS brings a much-desired file system feature: snapshots. A snapshot lets you freeze the state of a file system at a particular moment and continue to use and modify that file system while preserving the old data. It does so in a space-efficient fashion where, effectively, changes are tracked and only new data takes up additional space. This has the potential to be extremely valuable for backup by efficiently tracking the data that has changed since the last backup.

 ZFS includes snapshots and serialization mechanisms that make it efficient to backup file systems or transfer file systems to a remote location. Will APFS work like that? Probably not, answered Dominic Giampaolo, APFS lead developer. ZFS sends all changed data while Time Machine can have exclusion lists and the like. That seems surmountable, but we’ll see what Apple does. APFS right now is incompatible with Time Machine due to the lack of directory hard links, a fairly disgusting implementation that likely contributes to Time Machine’s questionable reliability. Hopefully APFS will create some efficient serialization for Time Machine backup.

 While Eric Tamura, APFS dev manager, demonstrated snapshots at WWDC, the required utilities aren’t included in the macOS Sierra beta. I used DTrace (technology I’m increasingly amazed that Apple ported from OpenSolaris) to find a tantalizingly-named new system call fs_snapshot; I’ll leave it to others to reverse engineer its proper use.

 Management

 APFS brings another new feature known as space sharing. A single APFS “container” that spans a device can have multiple “volumes” (file systems) within it. Apple contrasts this with the static allocation of disk space to support multiple HFS+ instances, which seems both specious and an uncommon use case. Both ZFS and btrfs have a similar concept of a shared pool of storage with nested file systems for administration and management.

 Speaking with Dominic and other members of the APFS team, we discussed how volumes are the unit by which users can control things like snapshots and encryption. You’d want multiple volumes to correspond with different policies around those settings. For example while you might want to snapshot and backup your system each day, the massive /private/var/vm/sleepimage (for saving memory when hibernating) should live on its own and not be backed up.

 Space sharing is more like an operational detail than a game changing feature. You can think of it like special folders with snapshot and encryption controls… which is probably why Apple’s marketing department has yet to make me a job offer. Unfortunately this feature isn’t working in the macOS Sierra beta, so I wasn’t able to have more than one volume per container. Adding new volumes can fail with an opaque error (-69625 mean anything to you?), but using a larger disk image resolve the problem.

 Next in this series: Space Efficiency and Clones

 Posted on June 19, 2016 at 7:34 pm by ahl · Permalink · 6 Comments
 In: Software · Tagged with: APFS

 APFS in Detail: Overview

 Apple announced a new file system that will make its way into all of its OS variants (macOS, tvOS, iOS, watchOS) in the coming years. Media coverage to this point has been mostly breathless elongations of Apple’s developer documentation. With a dearth of detail I decided to attend the presentation and Q&A with the APFS team at WWDC. Dominic Giampaolo and Eric Tamura, two members of the APFS team, gave an overview to a packed room; along with other members of the team, they patiently answered questions later in the day. With those data points and some first hand usage I wanted to provide an overview and analysis both as a user of Apple-ecosystem products and as a long-time operating system and file system developer.

 I’ve divided my review into several sections that span a few posts. I’d encourage you to jump around to topics of interest or skip right to the conclusion (or to the tweet summary). Highest praise goes to encryption; ire to data integrity.

 * Basics
 * Paying Down Debt
 * Encryption
 * Snapshots and Backup
 * Management and Space Sharing
 * Space Efficiency and Clones
 * Performance
 * Data Integrity
 * Conclusion

 Basics

 APFS, the Apple File System, was itself started in 2014 with Dominic as its lead engineer. It’s a stand-alone, from-scratch implementation (an earlier version of this post noted a dependency on Core Storage, but Dominic set me straight). I asked him about looking for inspiration in other modern file systems such as BSD’s HAMMER, Linux’s btrfs, or OpenZFS (Solaris, illumos, FreeBSD, Mac OS X, Ubuntu Linux, etc.), all of which have features similar to what APFS intends to deliver. (And note that Apple built a fairly complete port of ZFS, though Dominic was not apparently part of the group advocating for it.) Dominic explained that while, as a self-described file system guy (he built the file system in BeOS, unfairly relegated to obscurity when Apple opted to purchase NeXTSTEP instead), he was aware of them, but didn’t delve too deeply for fear, he said, of tainting himself.

 Dominic praised the APFS testing team as being exemplary. This is absolutely critical. A common adage is that it takes a decade to mature a file system. And my experience with ZFS more or less confirms this. Apple will be delivering APFS broadly with 3-4 years of development so will need to accelerate quickly to maturity.

 Paying Down Debt

 HFS was introduced in 1985 when the Mac 512K (of memory! Holy smokes!) was Apple’s flagship. HFS+, a significant iteration, shipped in 1998 on the G3 PowerMacs with 4GB hard drives. Since then storage capacities have increased by factors of 1,000,000 and 1,000 respectively. HFS+ has been pulled in a bunch of competing directions with different forks for different devices (e.g. the iOS team created their own HFS variant, working so covertly that not even the Mac OS team knew) and different features (e.g. journaling, case insensitive). It’s old; it’s a mess; and, critically, it’s missing a bunch of features that are really considered the basic cost of doing business for most operating systems. Wikipedia lists nanosecond timestamps, checksums, snapshots, and sparse file support among those missing features. Add to that the obvious gap of large device support and you’ve got a big chunk of the APFS feature list.

 APFS first and foremost pays down the unsustainable technical debt that Apple has been carrying in HFS+. (In 2001 ZFS grew from a similar need where UFS had been evolved since 1977.) It unifies the multifarious forks. It introduces the expected features. In general it first brings the derelict building up to code.

 Compression is an obvious gap in the APFS feature list that is common in many file systems. It’s conceptually quite easy, I told the development team (we had it in ZFS from the outset), so why not include it? To appeal to Dominic’s BeOS nostalgia I even recalled my job interview with Be in 2000 when they talked about how compression actually improved overall performance since data I/O is far more expensive than computation (obvious now, but novel then). The Apple folks agreed, and—in typical Apple fashion—neither confirmed nor denied while strongly implying that it’s definitely a feature we can expect in APFS. I’ll be surprised if compression isn’t included in its public launch.

 Next in this series: Encryption, Snapshots, and Backup

 Posted on June 19, 2016 at 7:29 pm by ahl · Permalink · 42 Comments
 In: Software · Tagged with: APFS
 « Older Entries
 * Follow @ahl

 * Recent Posts

 + From Prometheus to Sisyphus
 + DTrace at Home
 + A Filesystem on Noms
 + I Love Go; I Hate Go
 + APFS in Detail: Conclusions
 + APFS in Detail: Data Integrity
 + APFS in Detail: Performance
 + APFS in Detail: Space Efficiency and Clones
 + APFS in Detail: Encryption, Snapshots, and Backup
 + APFS in Detail: Overview
 + ZFS: Apple’s New Filesystem That Wasn’t
 + Finding What’s Next
 + Big News for ZFS on Linux
 + Delphix Technology Scholarship for Women
 + Delphix Sync
 + I am not a resource
 + First Rust Program Pain (So you can avoid it…)
 + On Blogging (Briefly)
 + DTrace OEL Dynamic Language Support
 + Delphix Week of Giving

 * Archives

 + February 2017 (1)
 + December 2016 (1)
 + August 2016 (2)
 + June 2016 (7)
 + May 2016 (1)
 + March 2016 (1)
 + November 2015 (2)
 + September 2015 (1)
 + June 2015 (1)
 + March 2015 (1)
 + December 2014 (2)
 + August 2014 (1)
 + June 2014 (2)
 + February 2014 (2)
 + December 2013 (1)
 + September 2013 (2)
 + August 2013 (1)
 + May 2013 (1)
 + February 2013 (2)
 + December 2012 (1)
 + November 2012 (2)
 + October 2012 (1)
 + September 2012 (1)
 + July 2012 (1)
 + April 2012 (2)
 + February 2012 (3)
 + January 2012 (1)
 + December 2011 (1)
 + November 2011 (1)
 + October 2011 (4)
 + September 2011 (1)
 + July 2011 (1)
 + June 2011 (1)
 + May 2011 (2)
 + November 2010 (4)
 + September 2010 (1)
 + August 2010 (5)
 + July 2010 (2)
 + March 2010 (1)
 + December 2009 (2)
 + September 2009 (1)
 + August 2009 (1)
 + July 2009 (1)
 + May 2009 (3)
 + April 2009 (1)
 + March 2009 (4)
 + February 2009 (3)
 + December 2008 (1)
 + November 2008 (2)
 + October 2008 (2)
 + August 2008 (1)
 + July 2008 (2)
 + June 2008 (2)
 + May 2008 (1)
 + April 2008 (2)
 + March 2008 (1)
 + January 2008 (1)
 + October 2007 (1)
 + August 2007 (2)
 + July 2007 (3)
 + May 2007 (1)
 + March 2007 (2)
 + January 2007 (2)
 + December 2006 (1)
 + November 2006 (1)
 + September 2006 (2)
 + August 2006 (1)
 + June 2006 (2)
 + May 2006 (4)
 + February 2006 (2)
 + December 2005 (1)
 + November 2005 (1)
 + October 2005 (5)
 + September 2005 (1)
 + August 2005 (3)
 + July 2005 (2)
 + June 2005 (7)
 + May 2005 (2)
 + April 2005 (2)
 + March 2005 (2)
 + February 2005 (1)
 + January 2005 (1)
 + November 2004 (5)
 + October 2004 (5)
 + August 2004 (6)
 + July 2004 (21)
 + June 2004 (5)

 * Tags

 APFS Apple brandz BryanCantrill CACM ChrisGeorge Compellent DDRdrive Delphix DTrace FFMF Fishworks flash GaloisField GeorgeWilson HSP illumos JedYueh JeffBonwick Joyent L2ARC Linux Mac OS X MattAhrens NetApp OEL OpenSolaris OpenStorageSummit OpenZFS Oracle Performance RAID RAID-Z raidz2 raidz3 software Solaris SSD Sun TheRegister TiVo virtualization VMware ZFS ZIL

 * Meta

 + Log in
 + Entries RSS
 + Comments RSS
 + WordPress.org
 © Adam Leventhal's blog.
 Powered by WordPress and Grey Matter.
From Prometheus to Sisyphus 
 
 When Apple announced their new file system, APFS, in June, I hustled to be in the front row of the WWDC presentation, questions with the presenters, and then the open Q&amp;A session. I took a week to write up my notes which turned into as 12 page behemoth of a blog post — longer than my college thesis. Despite reassurances from the tweeps, I was sure that the blog post was an order of magnitude longer than the modern attention span. I was wrong; so wrong that Ars Technica wanted to republish the blog post. Never underestimate the interest in all things Apple.
In that piece I left one big thread dangling. Apple shipped APFS as a technology preview, but they left out access to one of the biggest new features: snapshots. Digging around I noticed that there was a curiously named new system call, “fs_snapshot”, but explicitly didn’t investigate: the post was already too long (I thought), I had spent enough time on it, and someone else (surely! surely?) would want to pull on that thread.
Slow News Day
Every so often I’d poke around for APFS news, but there was very little new. Last month folks discovered that APFS was coming to iOS sooner rather than later. But there wasn’t anything new to play with or any revelations on how APFS would work.
I would search for “APFS snapshots”, “fs_snapshot”, anything I could think of to see if anyone had figured out how to make snapshots work on APFS. Nothing.
A few weekends ago, I decided to yank on that thread myself.
Prometheus
I started from the system call, wandered through Apple’s open source kernel, leaned heavily on DTrace, and eventually figured it out. Apple had shipped snapshots in APFS, they just hadn’t made it easy to get there. The folks at Ars were excited for a follow-up, and my investigation turned into this: “Testing out snapshots in Apple’s next-generation APFS file system.” 
Snapshots were there; the APIs were laid bare; I was going to bring fire to all the Mac fans; John Siracusa and Andy Ihnatko would carry me on their shoulders down the streets of the Internet.
Sisyphus
On the eve that this new piece was about to run I was nervously scrolling through Twitter as I took the bus home from work. Now that I had invested the time to research and write-up APFS snapshots I didn’t want someone else beating me to the punch.
Then I found this and my heart dropped:
Slides from the “Storing our digital lives: Mac filesystems from MFS to APFS” session at MacTech Conference 2016: https://t.co/uJJuqLL8n9 
— Rich Trouton (@rtrouton) November 17, 2016  
Skim past the craziness of MFS and the hairball of HFS, and start digging through the APFS section. Slide 49, “APFS Snapshots” and there it is “apfs_snapshot” — not a tool that anyone laboriously reverse engineered, deciphering system calls and semi-published APIs — a tool shipped from Apple and included in macOS by default. F — .
Apple had secreted this utility away (along with some others) in /System/Library/Filesystems/apfs.fs/Contents/Resources/ 
What To Do?
The article that was initially about a glorious act of discovery had become an article about the reinvention of the wheel. Conversely, vanishingly few people would recognize this as rediscovery since the apfs_snapshot tool was so obscure (9 hits on Google!).
We toned down the already modest chest-thumping and published the article this morning to a pretty nice response so far. I might have happier as an FAKE NEWS Prometheus, blissfully unaware of the pre-existence of fire, but I would have been mortified when the inevitable commenter, one of the few who had used apfs_snapshot, crushed me with my own boulder.
 	
 	
 	 DTrace at Home 
 
 I had been procrastinating making the family holiday card. It was a combination of having a lot on my plate and dreading the formulation of our annual note recapping the year; there were some great moments, but I’m glad I don’t have to do 2016 again. It was just before midnight and either I’d make the card that night or leave an empty space on our friends’ refrigerators. Adobe Illustrator had other ideas:
  
I’m not the first person to hit this. The problem seems to have existed since CS6 was released in 2016. None of the solutions was working for me, and — inspired by Sara Mauskopf’s excellent post — I was rapidly running out of the time bounds for the project. Enough; I’d just DTrace it.
A colleague scoffed the other day, “I mean, how often do you actually use DTrace?” In his mind DTrace was for big systems, critical system, when dollars and lives were at stake. My reply: I use DTrace every day. I can’t imagine developing software without DTrace, and I use it when my laptop (not infrequently) does something inexplicable (I’m forever grateful to the Apple team that ported it to Mac OS X).
First I wanted to make sure I had the name of the Illustrator process right:
$ sudo dtrace -n ‘syscall:::entry{ @[execname] = count(); }’
dtrace: description ‘syscall:::entry’ matched 500 probes
^C
pboard 1
watchdogd 2
awdd 3
...
com.apple.WebKit 7065
Google Chrome He 7128
Google Chrome 8099
Adobe Illustrato 36674
Glad I checked: “Adobe Illustrato”. Now we can be pretty sure that Illustrator is failing on setrlimit(2) and blowing up as result. Let’s confirm that it is in fact returning -1:
$ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato"/{ printf("%d %d", arg1, errno); }'
dtrace: description 'syscall::setrlimit:return' matched 1 probe
CPU ID FUNCTION:NAME
 0 532 setrlimit:return -1 1
There it is. And setrlimit(2) is failing with errno 1 which is EPERM (value too high for non-root user). I already tuned up the files limit pretty high. Let’s confirm that it is in fact setting the files limit and check the value to which it’s being set. To write this script I looked at the documentation for setrlimit(2) (hooray for man pages!) to determine that the position of the resource parameter (arg0) and the type of the value parameter (struct rlimit). I needed the DTrace copyin() subroutine to grab the structure from the process’s address space:
$ sudo dtrace -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this-&gt;r = *(struct rlimit *)copyin(arg1, sizeof (struct rlimit)); printf("%x %x %x", arg0, this-&gt;r.rlim_cur, this-&gt;r.rlim_max); }'
dtrace: description 'syscall::setrlimit:entry' matched 1 probe
CPU ID FUNCTION:NAME
 0 531 setrlimit:entry 1008 2800 7fffffffffffffff
Looking through /usr/include/sys/resource.h we can see that 1008 corresponds to the number of files (RLIMIT_NOFILE | _RLIMIT_POSIX_FLAG). Illustrator is trying to set that value to 0x7fffffffffffffff or 2⁶³-1. Apparently too big; I filed any latent curiosity for another day.
The quickest solution was to use DTrace again to whack a smaller number into that struct rlimit. Easy:
$ sudo dtrace -w -n 'syscall::setrlimit:entry/execname == "Adobe Illustrato"/{ this-&gt;i = (rlim_t *)alloca(sizeof (rlim_t)); *this-&gt;i = 10000; copyout(this-&gt;i, arg1 + sizeof (rlim_t), sizeof (rlim_t)); }'
dtrace: description 'syscall::setrlimit:entry' matched 1 probe
dtrace: could not enable tracing: Permission denied
Oh right. Thank you SIP. This isa new laptop (at least a new motherboard due to some bizarre issue) which probably contributed to Illustrator not working when once it did. Because it’s new I haven’t yet disabled the part of SIP that prevents you from using DTrace on the kernel or in destructive mode (e.g. copyout()). It’s easy enough to disable, but I’m reboot-phobic — I hate having to restart my terminals — so I went to plan B: lldb.
First I used DTrace to find the code that was calling setrlimit(2): using some knowledge of the x86 ISA/ABI:
$ sudo dtrace -n 'syscall::setrlimit:return/execname == "Adobe Illustrato" &amp;&amp; arg1 == -1/{ printf("%x", *(uintptr_t *)copyin(uregs[R_RSP], sizeof (uintptr_t)) - 5) }'
dtrace: description 'syscall::setrlimit:return' matched 1 probe
CPU ID FUNCTION:NAME
 0 532 setrlimit:return 1006e5b72
 0 532 setrlimit:return 1006e5b72
I ran it a few times to confirm the address of the call instruction and to make sure the location wasn’t being randomized. If I wasn’t in a rush I might have patched the binary, but Apple’s Mach-O Object format always confuses me. Instead I used lldb to replace the call with a store of 0 to %eax (to evince a successful return value) and some nops as padding (hex values I remember due to personal deficiencies):
(lldb) break set -n _init
Breakpoint 1: 47 locations.
(lldb) run
...
(lldb) di -s 0x1006e5b72 -c 1
0x1006e5b72: callq 0x1011628e0 ; symbol stub for: setrlimit
(lldb) memory write 0x1006e5b72 0x31 0xc0 0x90 0x90 0x90
(lldb) di -s 0x1006e5b72 -c 4
0x1006e5b72: xorl %eax, %eax
0x1006e5b74: nop
0x1006e5b75: nop
0x1006e5b76: nop
Next I just process detach and got on with making that holiday card…
DTrace Every Day
DTrace was designed for solving hard problems on critical systems, but the need to understand how systems behave exists in development and on consumer systems. Just because you didn’t write a program doesn’t mean you can’t fix it.
 	
 	
 	 A Filesystem on Noms 
 
 Since Noms dropped last week the dev community has seemed into it. “Git for data” — it simultaneously evokes something very familiar and yet unconstrained. Something that hasn’t been well-noted is how much care the team has taken to make Noms fun to build with, and it is.
  Noms is a content-addressable, decentralized, append-only database. It borrows concepts from a variety of interesting data systems. Obviously databases are represented: Noms is a persistent, transactional data repository. You can also see the fundamentals of git and other decentralized source code management tools. Noms builds up a chain of commits; those chains can be extended, forked, and shared, while historical data are preserved. Noms shares much in common with modern filesystems such as ZFS, btrfs, or Apple’s forthcoming APFS. Like those filesystems, Noms uses copy-on-write, never modifying data in situ; it forms a self-validating hierarchy of data; and it (intrinsically) supports snapshots and efficient movement of data between snapshots.
After learning a bit about Noms I thought it would be interesting to use it as the foundation for a filesystem. I wanted to learn about Noms, and contribute a different sort of example that might push the project in new and valuable ways. The Noms founders, Aaron and Raf, were fired up so I dove in…
What’s Modern?
When people talk about a filesystem being “modern” there’s a list of features that they often have in mind. Let’s look at how the Noms database stacks up:
Snapshots
A filesystem snapshot preserves the state of the filesystem for some future use — typically data recovery or fast cloning. Since Noms is append-only, every version is preserved. Snapshots are, therefore, a natural side effect. You can make a Noms “snapshot” — any commit in a dataset’s history — writeable by syncing it to a new dataset. Easy.
Dedup
The essence of dedup is that unique data should be stored exactly once. If you duplicate a file, a folder, or an entire filesystem the storage consumption should be close to zero. Noms is content addressable, unique data is only ever stored once. Every Noms dataset intrinsically removes duplicated data.
Consistency
A feature of a filesystem — arguably the feature of a filesystem — is that it shouldn’t ever lose or corrupt your data. One common technique to ensure consistency is to write new data to a new location rather than overwriting old data — so called copy-on-write (COW). Noms is append-only, it doesn’t throw out (or overwrite) old data; copying modified is required and explicit. Noms also recursively checksums all data — a feature of ZFS and btrfs, notably absent from APFS.
Backup
The ability to backup your data from a filesystem is almost as important as keeping it intact in the first place. ZFS, for example, lets you efficiently serialize and send the latest changes between systems. When pulling or pushing changes git also efficiently serializes just the changed data. Noms does something similar with its structured data. Data differences are efficiently computed to optimize for minimal data transfer.
Noms has all the core components of a modern filesystem. My goal was to write the translation layer to expose filesystem semantics on top of the Noms interfaces.
Designing a Schema
Initially, Badly
It’s in the name: Noms eats all the data. Feed it whatever data you like, and let Noms infer a schema as you go. For a filesystem though I wanted to define a fixed structure. I started with a schema modeled on a simplified ZFS. Filesystems keep track of files and directories with a structure called an “inode” each of which has a unique integer identifier, the “inode number”. ZFS keeps track of files and directories with DMU objects named by their integer ID. The schema would use a Map&lt;number, Inode&gt; to serve the same function (spoiler: read on and don’t copy this!):
struct Filesystem {
 inodes: Map&lt;Number, struct Inode {
 attr: struct Attr { /* e.g. permissions, modification time, etc. */ }
 contents: Union {
 struct File { data: Ref /* Noms pile of bytes */ } |
 struct Directory { contents: Map&lt;String, Number&gt; }
 }
 }&gt;
 rootInode: Number
 maxInode: Number
}
Nice and simple. Files are just Noms Blobs represented by a sequence of bytes. Directories are a Map of strings (the name of the directory entry) to the inode number; the inode number can be used to find the actual content by looking in the top-level map.
Schema philosophy
This made sense for a filesystem. Did it make sense for Noms? I wasn’t trying to put the APFS team out of work, rather I was creating a portal from the shell or Finder into Noms. To evaluate the schema, I had the benefit of direct access to the Noms team (and so can all developers at http://slack.noms.io/). I learned two guiding principles for data in Noms:
 Noms data should be self-validating. As much as possible the application should rely on noms to ensure consistency. My schema failed this test because the relationship between inode numbers contained in directories and the entires of the inodes map was something my code alone could maintain and validate.
 Noms data should be deterministic. A given collection of data should have a single representation; the Noms structures should be path and history independent. Two apparently identical datasets should be identical in the eyes of Noms to support efficient storage and transfer, and identical data should produce an identical hash at the root of the dataset. Here, again, my schema fell short because the inode number assigned to a given file or directory depended on how other objects were created. Two different users with two identical filesystems wouldn’t be able to simply sync the way they would with two identical git branches.
A Better Schema
My first try made for a fine filesystem, just not a Noms filesystem. With a better understanding of the principles, and with help from the Noms team, I built this schema:
struct Filesystem {
 root: struct Inode {
 attr: struct Attr { /* e.g. permissions, modification time, etc. */ }
 contents: Union {
 struct File { data: Ref&lt;Blob&gt; /* Noms pile of bytes */ } |
 struct Directory: { contents: Map&lt;string, Cycle&lt;1&gt;&gt; }
 }
 }
}
Obviously simpler; the thing that bears explanation is the use of so-called “Cycle” types. A Cycle is a means of expressing a recursive relationship within Noms types. The integer parameter specifies the ancestor struct to which the cycle refers. Consider a linked list type:
struct LinkedList {
 data: Blob
 next: Cycle&lt;0&gt;
}
The “next” field refers to immediately containing struct, LinkedList. In our filesystem schema, a Directory’s contents are represented by a map of strings (directory entry names) to Inodes, Cycle&lt;1&gt; referring to the struct “above” or “containing” the Directory type. (Read on for a visualization of this.)
Writing It
To build the filesystem I picked a FUSE binding for Go, dug into the Noms APIs, and wrestled my way through some Go heartache.
Working with Noms requires a slightly different mindset than other data stores. Recall in particular that Noms data is immutable. Adding a new entry into a Map creates a new Map. Setting a member of a Struct creates a new Struct. Changing nested structures such as the one defined by our schema requires unzipping it, and then zipping it back together. Here’s a Go snippet that demonstrates that methodology for creating a new directory:
Demo
Showing it off has all the normal glory of a systems demo! Check out the documentation for requirements.
Create and mount a filesystem from a new local Noms dataset:
$ go build
$ mkdir /var/tmp/mnt
$ go run nomsfs.go /var/tmp/nomsfs::fs /var/tmp/mnt
running...
You can open the folder and drop data into it.
     Your database fell into my filesystem! 
Now let’s take a look at the underlying Noms dataset:
$ noms show http://demo.noms.io/ahl_blog::fs
struct Commit {
 meta: struct {},
 parents: Set&lt;Ref&lt;Cycle&lt;0&gt;&gt;&gt;,
 value: struct Filesystem {
 root: struct Inode {
 attr: struct Attr {
 ctime: Number,
 gid: Number,
 mode: Number,
 mtime: Number,
 uid: Number,
 xattr: Map&lt;String, Blob&gt;,
 },
 contents: struct Directory {
 entries: Map&lt;String, Cycle&lt;1&gt;&gt;,
 } | struct Symlink {
 targetPath: String,
 } | struct File {
 data: Ref&lt;Blob&gt;,
 },
 },
 },
}({
 meta: {},
 parents: {
 5v82rie0be68915n1q7pmcdi54i9tmgs,
 },
 value: Filesystem {
 root: Inode {
 attr: Attr {
 ctime: 1.4705227450393803e+09,
 gid: 502,
 mode: 511,
 mtime: 1.4705227450393803e+09,
 uid: 110853,
 xattr: {},
 },
 contents: Directory {
 entries: {
 "usenix_winter91_faulkner.pdf": Inode {
 attr: Attr {
 ctime: 1.4705228859273868e+09,
 gid: 502,
 mode: 420,
 mtime: 1.468425783e+09,
 uid: 110853,
 xattr: {
 "com.apple.FinderInfo": 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 // 32 B
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00,
 "com.apple.quarantine": 30 30 30 31 3b 35 37 38 36 36 36 33 37 3b 53 61 // 21 B
 66 61 72 69 3b,
 },
 },
 contents: File {
 data: dmc45152ie46mn3ls92vvhnm41ianehn,
 },
 },
 },
 },
 },
 },
})
You can see the type at the top and then the actual filesystem contents. Let’s look at more complicated example where I’ve taken part of the Noms source tree and copied it to nomsfs. You can navigate around its structure courtesy of the Splore utility (take particular note of nested directories that show the recursive data definition described above):
 Embedded ‘Splore! http://splore.noms.io/?db=https://demo.noms.io/ahl_blog&amp;hash=2nhi5utm4s38hka22vt9ilv5i3l8r2ol  
You can see the all of the various states that the filesystem has been through — each state change — using noms log http://demo.noms.io/ahl_blog::fsnoms. You can sync it to your local computer with noms sync http://demo.noms.io/ahl_blog::fsnoms /var/tmp/fs or checkout some previous state from the log (just like a filesystem snapshot). Diff two states from the log or make your own changes and diff it with the original using noms diff.
Nom Nom Nom
It took less than 1000 lines of Go code to make Noms appear as a Window in the Finder, eating data as quickly as I could drag and drop (try it!). Imagine what Noms might look like behind other known data interfaces; it could bring git semantics to existing islands of data. Noms could form the basis of a new type of data lake — maybe one that’s simple and powerful enough to bring real results. Beyond the marquee features, Noms is fun to build with. I’m already working on my next Noms application.
 	
 	
 	 I Love Go; I Hate Go 
 
 I liked Go right away. It was close enough to C and Java to be instantly familiar, the examples and tutorials were straightforward, and I was quickly writing real code. I’ve wanted to learn Go since its popularity was surging few years ago. In no danger of being judged an early adopter, I happily found a great project that—as it happened—had to be in Go (more in a future post).
I Love Go
My first priority was not looking stupid. The folks I’d be doing this for are actual Go developers; I wanted my code to fit in without imposing with too many questions. They had no style guide. Knowing that my 80-column width sensibilities expose an unattractive nostalgia, I went looking for a max line length. There wasn’t one. Then I discovered gofmt, the simple tool that Go employs to liberate developers from the tyranny of stylistic choice. It takes Go code and spits it back out in the One True Style. I love it. I was raised in an engineering culture with an exacting style guide, but any style guide has gaps. Factions form; style-originalists face off against those who view (incorrectly!) the guide as a living document. I updated my decades-old .vimrc to gofmt on save. These Go tyrants were feeling like my kind of tyrant.
One of the things that turned me off of C++ (98, 11, and 14) is its increasing amount of magic. Go is comparatively straightforward. When I reach for something I find that it’s where I expect it to be. Libraries and examples aren’t mysterious. Error messages are non-mysterious (other than quickly resolved confusion about methods with “pointer receivers”). Contrast this with Rust whose errors read like mind-bindingly inscrutable tax forms.
Go’s containment-based inheritance is easy enough to reason about. Its interfaces are similarly no-nonsense and pragmatic. You don’t have to define a structure as implementing an interface. You can use an interface to describe anything that implements it. Simple. This is particularly useful for testing. In a messy environment with components beyond your control you can apply interfaces to other people’s code, and then mock it out.
The toolchain is, again, simple to use—the benefit of starting from scratch—and makes for fast compilation, quick testing, and easy integration with a good-sized ecosystem. I stopped worrying about dependencies, rebuilding, etc. knowing that go run would find errors wherever I had introduced them and do so remarkably quickly.
I Hate Go
Go is opinionated. Most successful products have that strong sense of what they are and what they aren’t; Go draws as sharp a line as any language. I was seduced by its rightheadedness around style, but with anything or anyone that opinionated you’ll find some of those opinions weird and others simply off-putting.
Reading the official documentation, I found myself in the middle of a section prefixed with the phrase “if GOOS is set to plan9”. Wow. I’m a few standard deviations from the norm in terms of being an OS nerd, but I’ve never even seen Plan 9. I knew that the Plan 9 folks got the band back together to create Go; it’s great that their pop audiences don’t dissuade them from playing off their old B-sides. Quirky, but there’s nothing wrong with that.
I wanted to check an invariant; how does Go do assertions? Fuck you, you’re a bad programmer. That’s how. The Go authors feel so strongly about asserts being typically misused that they refuse to provide them. So folks use one (or more) of some workable libraries.
I created an infinite recursion, overflowing the stack. Go produces the first 100 stack frames and that’s it. Maybe you can change that, but I couldn’t figure out how. (“go stackoverflow” is about the most useless thing you can search for; chapeau, Go and Stackoverflow respectively.) I could be convinced that I only want 100 stack frames, but not the last 100, not the same 4 over and over again. I ended up limiting the stack size with runtime.debug.SetMaxStack(), Goldilocks-ing it between too big to catch the relevant frames and too small to allow for normal operation.
I tried using other tools (ahem, DTrace) to print the stack, but, of course, the Go compiler omits frame pointers rendering the stacks unobservable to debuggers. Ditto arguments due to ABI-non-compliant calling conventions, but that’s an aside. The environment variable GOEXPERIMENT=framepointer is supposed to compile with frame pointers, but it was a challenge to rebuild the world. All paths seemed to lead me to my former-colleague’s scathing synopsis: Golang is Trash.
As fun as it is to write code in Go, debugging in Go is no fun at all. I may well just be ignorant of the right tooling. But there sure isn’t a debugger with the simple charm of go build for compilation, go test for testing, or go run for execution.
Immaturity and Promise
Have you ever been in a relationship where minor disagreements immediately escalated to “maybe we should break up?” The Go documentation even seems ready to push you to some other language at the slightest affront. Could I have asserts? Sure, if you’re a bad programmer. Perhaps ABI compliance has its merits? I’m sure you could find that in some other language. Could you give me the absolute value of this int? Is something wrong with your ‘less than’ key?
I’m sure time will, as it tends to, bring pragmatism. I appreciate that Go does have strong opinions (it’s just hard to remember that when I disagree with them). Weak opinions are what turn languages into unreadable mishmashes of overlapping mechanism. My favorite example of this is Perl. My first real programming job was in Perl. I was the most avid reader teenage of the Perl llama and camel books. Ask me about my chat with Larry Wall, Perl’s creator, if you see a beer in my hand. In an interview, Larry said, “In Perl 6, we decided it would be better to fix the language than fix the user”. Contrast this with Go’s statement on assertions:
 Go doesn’t provide assertions. They are undeniably convenient, but our experience has been that programmers use them as a crutch to avoid thinking about proper error handling and reporting. 
Perl wants to be whatever the user wants to be. The consumate pleaser. Go sees the user as flawed and the strictures of the language as the cure. It’s an authoritarian, steadfast in its ideals, yet too sensitive to find compromise (sound like anyone we all know?).
Despite my frustrations I really enjoy writing code in Go. It’s clean and supported by a great community and ecosystem. I’m particularly heartened that Go 1.7 will compile with frame pointers by default. Diagnosing certain types of bugs is still a pain in the neck, but I’m sure it will improve and I’ll see where I can pitch in.
 	
 	
 	 APFS in Detail: Conclusions 
 
  This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the first post for the table of contents. 
Summing Up
I’m not sure Apple absolutely had to replace HFS+, but likely they had passed an inflection point where continuing to maintain and evolve the 30+ year old software was more expensive than building something new. APFS is a product born of that assessment.
Based on what Apple has shown I’d surmise that its core design goals were:
satisfying all consumers (laptop, phone, watch, etc.)
encryption as a first-class citizen
snapshots for modernized backup.
Those are great goals that will benefit all Apple users, and based on the WWDC demos APFS seems to be on track (though the macOS Sierra beta isn’t quite as far along).
In the process of implementing a new file system the APFS team has added some expected features. HFS was built when 400KB floppies ruled the Earth (recognized now only as the ubiquitous and anachronistic save icon). Any file system started in 2014 should of course consider huge devices, and SSDs–check and check. Copy-on-write (COW) snapshots are the norm; making the Duplicate command in the Finder faster wasn’t much of a detour. The use case is unclear, it’s a classic garbage can theory solution, a solution in search of a problem, but it doesn’t hurt and it makes for a fun demo. The beach ball of doom earned its nickname; APFS was naturally built to avoid it.
There are some seemingly absent or ancillary design goals: performance, openness, and data integrity. Squeezing the most IOPS or throughput out of a device probably isn’t critical on watchOS, and it’s relevant only to a small percentage of macOS users. It will be interesting to see how APFS performs once it ships (measuring any earlier would only misinform the public and insult the APFS team). The APFS development docs have a bullet on open source: “An open source implementation is not available at this time.” I don’t expect APFS to be open source at this time or any other, but prove me wrong, Apple. If APFS becomes world-class I’d love to see it in Linux and FreeBSD–maybe Microsoft would even jettison their ReFS experiment. My experience with OpenZFS has shown that open source accelerates that path to excellence. It’s a shame that APFS lacks checksums for user data and doesn’t provide for data redundancy. Data integrity should be job one for a file system, and I believe that that’s true for a watch or phone as much as it is for a server.
At stability, APFS will be an improvement, for Apple users of all kinds, on every device. There are some clear wins and some missed opportunities. Now that APFS has been shared with the world the development team is probably listening. While Apple is clearly years past the decision to build from scratch rather than adopting existing modern technology, there’s time to raise the priority of data integrity and openness. I’m impressed by Apple’s goal of using APFS by default within 18 months. Regardless of how it goes, it will be an exciting transition.
 	
 	
 	 APFS in Detail: Data Integrity 
 
  This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the first post for the table of contents. 
Data Integrity
Arguably the most important job of a file system is preserving data integrity. Here’s my data, don’t lose it, don’t change it. If file systems could be trusted absolutely then the “only” reason for backup would be the idiot operators (i.e. you and me). There are a few mechanisms that file systems employ to keep data safe.
  
Redundancy
APFS makes no claims with regard to data redundancy. As Apple’s Eric Tamura noted at WWDC, most Apple devices have a single storage device (i.e. one logical SSD) making RAID, for example, moot. Instead redundancy comes from lower layers such as Apple RAID (apparently a thing), hardware RAID controllers, SANs, or even the “single” storage devices themselves..
As an aside note that SSDs in most Apple products where APFS will run include multiple more-or-less independent NAND chips. High-end SSDs do implement data redundancy within the device, but it comes at the price of reduced capacity and performance. As noted above, the “flash-optimization” of APFS doesn’t actually extend much below the surface of the standard block device interface, but the raw materials for innovation are there.
Also, APFS removes the most common way of a user achieving local data redundancy: copying files. A copied file in APFS actually creates a lightweight clone with no duplicated data. Corruption of the underlying device would mean that both “copies” were damaged whereas with full copies localized data corruption would affect just one.
  
Crash Consistency
Computer systems can fail at any time—crashes, bugs, power outages, etc.—so file systems need to anticipate and recover from these scenarios. The old old old school method is to plod along and then have a special utility to check and repair the file system during boot (fsck, short for file system check). More modern systems labor to achieve an always consistent format, or only narrow windows of inconsistency, obviating the need for the full, expensive fsck. ZFS, for example, builds up new state on disk and then atomically transitions from the previous state to the new one with a single atomic operation.
Overwriting data creates the most obvious opening for inconsistency. If the file system needs to overwrite several regions there is a window where some regions represent the new state and some represent the former state. Copy-on-write (COW) is a method to avoid this by always allocating new regions and then releasing old ones for reuse rather than modifying data in-place. APFS claims to implement a “novel copy-on-write metadata scheme”; APFS lead developer Dominic Giampaolo emphasized the novelty of this approach without delving into the details. In conversation later, he made it clear that APFS does not employ the ZFS mechanism of copying all metadata above changed user data which allows for a single, atomic update of the file system structure.
It’s surprising to see that APFS includes fsck_apfs—even after asking Dominic I’m not sure why it would be necessary. For comparison I don’t believe there’s been an instance where fsck for ZFS would have found a problem that the file system itself didn’t already know how to detect. But Dominic was just as confused about why ZFS would forego fsck, so perhaps it’s just a matter of opinion.
  
Checksums
Notably absent from the APFS intro talk was any mention of checksums. A checksum is a digest or summary of data used to detect (and correct) data errors. The story here is surprisingly nuanced. APFS checksums its own metadata but not user data. The justification for checksumming metadata is strong: there’s relatively not much of it (so the checksums don’t consume much storage) and losing metadata can cast a potentially huge shadow of data loss. If, for example, metadata for a top level directory is corrupted then potentially all data on the disk could be rendered inaccessible. ZFS duplicates metadata (and triple duplicates top-level metadata) for exactly this reason.
Explicitly not checksumming user data is a little more interesting. The APFS engineers I talked to cited strong ECC protection within Apple storage devices. Both flash SSDs and magnetic media HDDs use redundant data to detect and correct errors. The engineers contend that Apple devices basically don’t return bogus data. NAND uses extra data, e.g. 128 bytes per 4KB page, so that errors can be corrected and detected. (For reference, ZFS uses a fixed size 32 byte checksum for blocks ranging from 512 bytes to megabytes. That’s small by comparison, but bear in mind that the SSD’s ECC is required for the expected analog variances within the media.) The devices have a bit error rate that’s tiny enough to expect no errors over the device’s lifetime. In addition, there are other sources of device errors where a file system’s redundant check could be invaluable. SSDs have a multitude of components, and in volume consumer products they rarely contain end-to-end ECC protection leaving the possibility of data being corrupted in transit. Further, their complex firmware can (does) contain bugs that can result in data loss.
The Apple folks were quite interested in my experience with regard to bit rot (aging data silently losing integrity) and other device errors. I’ve seen many instances where devices raised no error but ZFS (correctly) detected corrupted data. Apple has some of the most stringent device qualification tests for its vendors; I trust that they really do procure the best components. Apple engineers I spoke with claimed that bit rot was not a problem for users of their devices, but if your software can’t detect errors then you have no idea how your devices really perform in the field. ZFS has found data corruption on multi-million dollar storage arrays; I would be surprised if it didn’t find errors coming from TLC (i.e. the cheapest) NAND chips in some of Apple’s devices. Recall the (fairly) recent brouhaha regarding storage problems in the high capacity iPhone 6. At least some of Apple’s devices have been imperfect.
As someone who has data he cares about on a Mac, who has seen data lost from HFS, and who knows that even expensive, enterprise-grade equipment can lose data, I would gladly sacrifice 16 bytes per 4KB–less than 1% of my device’s size.
  
Scrub
As data ages you might occasionally want to check for bit rot. Likely fsck_apfs can accomplish this; as noted though there’s no data redundancy and no checksums for user data, so scrub would only help to find problems and likely wouldn’t help to correct them. And if it makes it any easier for Apple to reverse course, let’s say it’s for the el cheap-o drive I bought from Fry’s not for the gold-plated device I got from Apple.
 Next in this series: Conclusions  
 	
 	
 	 APFS in Detail: Performance 
 
  This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the first post for the table of contents. 
Performance
APFS claims to be optimized for flash. Flash memory (NAND) is the stuff in your speedy SSD. Apple changed the computing industry when it put flash into the iPod and iPhone, volumes for which fundamentally changed the economics of flash. This consumer change impacted the enterprise (as it often does), giving rise to hybrid and all-flash arrays. Ten years ago flash cost as much as DRAM; now it’s challenging the economics of hard disks.
SSDs mimic the block interface of conventional hard drives, but the underlying technology is completely different. In particular while magnetic media can read or write sectors arbitrarily, flash erases large chunks (blocks) and reads and writes smaller chunks (pages). The management is done by what’s called the flash translation layer (FTL), software that makes blocks and pages appear more like a hard drive. An FTL is very similar to a file system, creating a virtual mapping (a translation) between block addresses and locations within the media. Apple controls the full stack including the SSD, FTL, and file system; they could have built something differentiated, optimizing this components to work together. What APFS does, however, is simply write in patterns known to be more easily handled by NAND. It’s a file system with flash-aware characteristics rather than one written explicitly for the native flash interfaces, more or less what you’d expect in 2016.
Also on the topic of flash, APFS includes TRIM support. TRIM is a command in the ATA protocol that allows a file system to indicate to an SSD (specifically, its FTL) that some space has been freed. SSDs require significant free space and perform better when there’s more of it; they include more physical space than they advertise. For example, my 1TB SSD includes 1TB (240 = 10244) bytes of flash but only reports 931GB of available space, sneakily matching the storage industry’s self-serving definition of 1TB (10004 = 1 trillion bytes). With more free space, FTLs can trade off space efficiency for performance and longevity. TRIM has become expected of file systems; it’s unsurprising that APFS supports it. The problem with TRIM though is that it’s only useful when there’s free space: it’s something of a benchmark special. Once your disk is mostly full (as mine are in my laptop and phone basically at all times) TRIM doesn’t do anything for you. I doubt that TRIM will bring any discernible benefit for APFS users beyond the placebo effect of feature parity.
APFS also focuses on latency; Apple’s number one goal is to avoid the beachball of doom. APFS addresses this with I/O QoS (quality of service) to prioritize accesses that are immediately visible to the user over background activity that doesn’t have the same time-constraints. This is inarguably a benefit to users and a sophisticated file system capability.
 Next in this series: Data Integrity  
 	
 	
 	 APFS in Detail: Space Efficiency and Clones 
 
  This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the first post for the table of contents. 
  
Space Efficiency
A modern trend in file systems has been to store data more efficiently to effectively increase the size of your device. Common approaches include compression (which, as noted above, is very very likely coming) and deduplication. Dedup finds common blocks and avoids storing them multiply. This is potentially highly beneficial for file servers where many users or many virtual machines might have copies of the same file; it’s probably not useful for the single-user or few-user environments that Apple cares about. (Yes, they have server-ish offerings but their heart clearly isn’t into it.) It’s also furiously hard to do well as I learned painfully while supporting ZFS.
Apple’s sort-of-unique contribution to space efficiency is constant time cloning of files and directories. As a quick aside, “files” in macOS are often really directories; it’s a convenient lie they tell to allow logically related collections of files to be treated as an indivisible unit. Right click an application and select “Show Package Contents” to see what I mean. Accordingly, I’m going to use the term “file” rather than “file or directory” in sympathy for the patient readers who have made it this far.
With APFS, if you copy a file within the same file system (or possibly the same container; more on this later), no data is actually duplicated. Instead a constant amount of metadata is updated and the on-disk data is shared. Changes to either copy cause new space to be allocated (so-called “copy on write” or COW).
I haven’t seen this offered in other file systems, and it clearly makes for a good demo, but it got me wondering about the use case (UPDATE: btrfs supports this and calls the feature “reflinks”–link by reference). Copying files between devices (e.g. to a USB stick for sharing) still takes time proportional to the amount of data copied of course. Why would I want to copy a file locally? The common case I could think of is the layman’s version control: “thesis”, “thesis-backup”, “thesis-old”, “thesis-saving because I’m making edits while drunk”.
There are basically three categories of files:
Files that are fully overwritten each time; images, MS Office docs, videos, etc.
Files that are appended to, mostly log files
Files with a record-based structure, such as database files.
For the average user, most files fall into that first category. So with APFS I can make a copy of my document and get the benefits of space sharing, but those benefits will be eradicated as soon as I save the new revision. Perhaps users of larger files have a greater need for this and have a better idea of how it might be used.
Personally, my only use case is taking a file, say time-shifted Game of Thrones episodes falling into the “fair use” section of copyright law, and sticking it in Dropbox. Currently I need to choose to make a copy or permanently move the file to my Dropbox folder. Clones would let me do this more easily. But then so would hard links (a nearly ubiquitous file system feature that lets a file appear in multiple directories).
Clones open the door for potential confusion. While copying a file may take up no space, so too deleting a file may free no space. Imagine trying to free space on your system, and needing to hunt down the last clone of a large file to actually get your space back.
  APFS engineers don’t seem to have many use cases in mind; at WWDC they asked for suggestions from the assembled developers (the best I’ve heard is for copied VMs; not exactly a mass-market problem). If the focus is generic revision control, I’m surprised that Apple didn’t shoot for a more elegant solution. One could imagine functionality with APFS that allows a user to enable per-file Time Machine, change tracking for any file. This would create a new type of file where each version is recorded transparently and automatically. You could navigate to previous versions, prune the history, or delete the whole pile of versions at once (with no stray clones to hunt down). In fact, Apple introduced something related 5 years ago, but I’ve literally never seen or heard of it until researching this post (show of hands if you’ve clicked “Browse All Versions…”). APFS could clean up its implementation, simplify its use, and bring generic support for all applications. None of this solves my Game of Thrones storage problem, but I’m not even sure it’s much of a problem…
Side note: Finder copy creates space-efficient clones, but cp from the command line does not.
 Next in this series: Performance  
 	
 	
 	 APFS in Detail: Encryption, Snapshots, and Backup 
 
  This series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the first post for the table of contents. 
  
Encryption
Encryption is clearly a core feature of APFS. This comes from diverse requirements from the various devices, for example multiple keys within file systems on the iPhone or per-user keys on laptops. I heard the term “innovative” quite a bit at WWDC, but here the term is aptly applied to APFS. It supports several different encryption choices for a file system:
Unencrypted
Single-key for metadata and user data
Multi-key with different choices for metadata, files, and even sections of a file (“extents”)
Multi-key encryption is particularly relevant for portables where all data might be encrypted, but unlocking your phone provides access to an additional key and therefore additional data. Unfortunately this doesn’t seem to be working in the first beta of macOS Sierra (specifying fileEncryption when creating a new volume with diskutil results in a file system that reports “Is Encrypted” as “No”).
Related to encryption, I noticed an undocumented feature while playing around with diskutil (which prompts you for interactive confirmation of the destructive power of APFS unless this is added to the command-line: -IHaveBeenWarnedThatAPFSIsPreReleaseAndThatIMayLoseData; I’m not making this up). APFS (apparently) supports constant time cryptographic file system erase, called “effaceable” in the diskutil output. This presumably builds a secret key that cannot be extracted from APFS and encrypts the file system with it. A secure erase then need only delete the key rather than needing to scramble and re-scramble the full disk to ensure total eradication. Various iOS docs refer to this capability requiring some specialized hardware; it will be interesting to see what the option means on macOS. Either way, let’s not mention this to the FBI or NSA, agreed?
  
Snapshots and Backup
APFS brings a much-desired file system feature: snapshots. A snapshot lets you freeze the state of a file system at a particular moment and continue to use and modify that file system while preserving the old data. It does so in a space-efficient fashion where, effectively, changes are tracked and only new data takes up additional space. This has the potential to be extremely valuable for backup by efficiently tracking the data that has changed since the last backup.
ZFS includes snapshots and serialization mechanisms that make it efficient to backup file systems or transfer file systems to a remote location. Will APFS work like that? Probably not, answered Dominic Giampaolo, APFS lead developer. ZFS sends all changed data while Time Machine can have exclusion lists and the like. That seems surmountable, but we’ll see what Apple does. APFS right now is incompatible with Time Machine due to the lack of directory hard links, a fairly disgusting implementation that likely contributes to Time Machine’s questionable reliability. Hopefully APFS will create some efficient serialization for Time Machine backup.
While Eric Tamura, APFS dev manager, demonstrated snapshots at WWDC, the required utilities aren’t included in the macOS Sierra beta. I used DTrace (technology I’m increasingly amazed that Apple ported from OpenSolaris) to find a tantalizingly-named new system call fs_snapshot; I’ll leave it to others to reverse engineer its proper use.
  
Management
APFS brings another new feature known as space sharing. A single APFS “container” that spans a device can have multiple “volumes” (file systems) within it. Apple contrasts this with the static allocation of disk space to support multiple HFS+ instances, which seems both specious and an uncommon use case. Both ZFS and btrfs have a similar concept of a shared pool of storage with nested file systems for administration and management.
Speaking with Dominic and other members of the APFS team, we discussed how volumes are the unit by which users can control things like snapshots and encryption. You’d want multiple volumes to correspond with different policies around those settings. For example while you might want to snapshot and backup your system each day, the massive /private/var/vm/sleepimage (for saving memory when hibernating) should live on its own and not be backed up.
Space sharing is more like an operational detail than a game changing feature. You can think of it like special folders with snapshot and encryption controls… which is probably why Apple’s marketing department has yet to make me a job offer. Unfortunately this feature isn’t working in the macOS Sierra beta, so I wasn’t able to have more than one volume per container. Adding new volumes can fail with an opaque error (-69625 mean anything to you?), but using a larger disk image resolve the problem.
 Next in this series: Space Efficiency and Clones  
 	
 	
 	 APFS in Detail: Overview 
 
   Apple announced a new file system that will make its way into all of its OS variants (macOS, tvOS, iOS, watchOS) in the coming years. Media coverage to this point has been mostly breathless elongations of Apple’s developer documentation. With a dearth of detail I decided to attend the presentation and Q&amp;A with the APFS team at WWDC. Dominic Giampaolo and Eric Tamura, two members of the APFS team, gave an overview to a packed room; along with other members of the team, they patiently answered questions later in the day. With those data points and some first hand usage I wanted to provide an overview and analysis both as a user of Apple-ecosystem products and as a long-time operating system and file system developer.
I’ve divided my review into several sections that span a few posts. I’d encourage you to jump around to topics of interest or skip right to the conclusion (or to the tweet summary). Highest praise goes to encryption; ire to data integrity.
 Basics 
 Paying Down Debt 
 Encryption 
 Snapshots and Backup 
 Management and Space Sharing 
 Space Efficiency and Clones 
 Performance 
 Data Integrity 
 Conclusion 
Basics
APFS, the Apple File System, was itself started in 2014 with Dominic as its lead engineer. It’s a stand-alone, from-scratch implementation (an earlier version of this post noted a dependency on Core Storage, but Dominic set me straight). I asked him about looking for inspiration in other modern file systems such as BSD’s HAMMER, Linux’s btrfs, or OpenZFS (Solaris, illumos, FreeBSD, Mac OS X, Ubuntu Linux, etc.), all of which have features similar to what APFS intends to deliver. (And note that Apple built a fairly complete port of ZFS, though Dominic was not apparently part of the group advocating for it.) Dominic explained that while, as a self-described file system guy (he built the file system in BeOS, unfairly relegated to obscurity when Apple opted to purchase NeXTSTEP instead), he was aware of them, but didn’t delve too deeply for fear, he said, of tainting himself.
Dominic praised the APFS testing team as being exemplary. This is absolutely critical. A common adage is that it takes a decade to mature a file system. And my experience with ZFS more or less confirms this. Apple will be delivering APFS broadly with 3-4 years of development so will need to accelerate quickly to maturity.
  
Paying Down Debt
HFS was introduced in 1985 when the Mac 512K (of memory! Holy smokes!) was Apple’s flagship. HFS+, a significant iteration, shipped in 1998 on the G3 PowerMacs with 4GB hard drives. Since then storage capacities have increased by factors of 1,000,000 and 1,000 respectively. HFS+ has been pulled in a bunch of competing directions with different forks for different devices (e.g. the iOS team created their own HFS variant, working so covertly that not even the Mac OS team knew) and different features (e.g. journaling, case insensitive). It’s old; it’s a mess; and, critically, it’s missing a bunch of features that are really considered the basic cost of doing business for most operating systems. Wikipedia lists nanosecond timestamps, checksums, snapshots, and sparse file support among those missing features. Add to that the obvious gap of large device support and you’ve got a big chunk of the APFS feature list.
APFS first and foremost pays down the unsustainable technical debt that Apple has been carrying in HFS+. (In 2001 ZFS grew from a similar need where UFS had been evolved since 1977.) It unifies the multifarious forks. It introduces the expected features. In general it first brings the derelict building up to code.
Compression is an obvious gap in the APFS feature list that is common in many file systems. It’s conceptually quite easy, I told the development team (we had it in ZFS from the outset), so why not include it? To appeal to Dominic’s BeOS nostalgia I even recalled my job interview with Be in 2000 when they talked about how compression actually improved overall performance since data I/O is far more expensive than computation (obvious now, but novel then). The Apple folks agreed, and—in typical Apple fashion—neither confirmed nor denied while strongly implying that it’s definitely a feature we can expect in APFS. I’ll be surprised if compression isn’t included in its public launch.
 Next in this series: Encryption, Snapshots, and Backup
Recent Posts
http://dtrace.org/blogs/ahl/
https://www.adamtuliper.com/
Adam Tuliper's Development Tips
2018-10-19
Adam Tuliper
<div><div>
<p>
ASP.NET 5 is a new version of ASP.NET that has been built from the ground up with performance in mind, it is modular, and supports multiple platforms such as Windows, OSX, and Linux. ASP.NET 5 no longer contains Web Forms, it is purely MVC based and supports both C# and VB.NET. This may be a difficult point for some. Going forward, there was a lot of baggage Web Forms carried around and it was an abstraction to the web, which isn't how most modern frameworks work. As such, MVC 6 is the only supported option in ASP.NET 5. You can however still create Web Forms apps in Visual Studio 2015 (for ex. on .NET 4.6), just not for ASP.NET 5.</p><p>
The WebAPI technology (that released back with MVC 4) has now been merged into a unified controller model in ASP.NET 5. ASP.NET 5 now has out of the box support for Dependency Injection without the  need for any third party DI containers (although <a href="https://github.com/aspnet/DependencyInjection">several major ones work</a> with it if that's your use case), and performance that can meet or exceed NodeJS applications. Let's not forget self hosting (running without installing into a web server) as well across all the supported platforms and of course you can host in IIS. You can be up and running self-hosting a RESTful api in minutes, <a href="https://msdn.microsoft.com/en-us/Library/vs/alm/Build/azure/deploy-aspnet5">publish to Azure</a> and support millions of users, or <a href="http://docs.asp.net/en/latest/publishing/iis.html">host in your web server</a>.</p><p>
ASP.NET 5 has a modular design so you only run the components that you need in your application. Prior versions of ASP.NET had significant overhead loading libraries that may have never been used. The memory footprint of ASP.NET 5 is significantly smaller and you can truly run your application now side by side with multiple versions of .NET and your application code.</p><p>
As stated in the <a href="http://docs.asp.net/en/latest/conceptual-overview/aspnet.html">docs</a> website</p><ul>
<li>New light-weight and modular HTTP request pipeline</li>
<li>Ability to host on IIS or self-host in your own process</li>
<li>Built on .NET Core, which supports true side-by-side app versioning</li>
<li>Ships entirely as NuGet packages</li>
<li>Integrated support for creating and using NuGet packages</li>
<li>Single aligned web stack for Web UI and Web APIs</li>
<li>Cloud-ready environment-based configuration</li>
<li>Built-in support for dependency injection</li>
<li>New tooling that simplifies modern web development</li>
<li>Build and run cross-platform ASP.NET apps on Windows, Mac and Linux</li>
<li>Open source and community focused</li>
</ul>
<ul>
</ul>
<p>ASP.NET 5 is now RC (Release Candidate) which means you will get support for it in production - although expect some further changes.<br />
<br /></p>
<h3>
What does ASP.NET 5 run on?</h3>
<p>
ASP.NET 5 <b><i>runs on</i></b> Windows, OSX, Linux, and even <a href="http://www.hanselman.com/blog/PublishingAnASPNET5AppToDockerOnLinuxWithVisualStudio.aspx">Docker containers</a>. It <b><i>loads under .</i></b>NET 4.5+, .NET 4.6, Mono, and .NET Core (more on that later).</p><p>
As a side note, there are some interesting future changes brewing here in determining what defines the .NET system your app will run on, and that is the <a href="https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/standard-platform.md">.NET Platform Standard </a>which defines a minimum set of base classes required to run your app.This is something to keep an eye on as it changes how our apps will refer to .NET to run.</p><h3>
Getting Started</h3><p>
First you need the bits for ASP.NET 5 which you can retrieve from <a href="http://get.asp.net/">http://get.asp.net</a>. You can use all of your favorite code editors to create ASP.NET 5 applications. My preference is Visual Studio 2015 or Visual Studio Code as it has all the tooling to understand your apps better, but it's fairly trivial to do this with Sublime, Notepad, Brackets, etc.</p><p>
<a href="//3.bp.blogspot.com/-006pN_WCuc8/Vnfb5cJAV4I/AAAAAAAABO8/7zx8HTB9UKU/s1600/editors.png"></a></p><p>
Next create an application. You have a couple options here. The first is in Visual Studio, do a File-New Project-&gt;Web, select ASP.NET Web Application. This launches then a new dialog where you could select the .NET 4.6.1 template for One ASP.NET (IE using MVC, Web API, and Web Forms all in one project) which I want to stress is not ASP.NET 5. You must select one of the ASP.NET 5 projects from the ASP.NET 5 section (makes sense, right?).</p><p>
<a href="//1.bp.blogspot.com/-yi0Diwbiyyk/Vnfcw56E0CI/AAAAAAAABPI/Wyn7EGPz6GI/s1600/aspnet%2Bnew%2Bproj%2B1.png"></a></p><p>
<a href="//1.bp.blogspot.com/-lqT2RNdRFkA/Vnfc8V62jPI/AAAAAAAABPU/TzCVLLoz3Lk/s1600/aspnet%2Bnew%2Bproject2.png"></a></p><h4>
Yeoman</h4>
<p><a href="//2.bp.blogspot.com/-oGPHombBK7A/VnfqNxwV8qI/AAAAAAAABQI/ri-VTjAJzoc/s1600/yeoman.png"></a></p>
<p>
If you like using Yeoman (no idea what Yeoman is? Check out our free video <a href="https://channel9.msdn.com/Series/Package-Management-and-Workflow-Automation/04#time=35m42s">here </a>on Microsoft Virtual Academy) you can use <a href="https://github.com/omnisharp/generator-aspnet#generator-aspnet">Yeoman generator for ASP.NET 5</a>. This will code-gen your base project for you. The commands (assuming you have NPM installed by installing NodeJS) are simply:</p><pre>npm install -g yo
npm install -g generator-aspnet
yo aspnet
</pre>
<p>
That will generate your project and you can simply right click on that generated folder and open in Visual Studio Code to see your entire project. There's actually a <a href="https://code.visualstudio.com/Docs/runtimes/ASPnet5">great article on this Visual Studio Code ASP.NET 5 workflow</a> already, so I won't repeat that here.<br />
<a href="//2.bp.blogspot.com/-kriKxR4rf5k/VnfehXd3cbI/AAAAAAAABPg/YkDu-Dsl2WQ/s1600/open%2Bwith%2Bcode.png"></a></p><h3>
Project.json and the Three Commands</h3><p>
There are three commands used in managing your ASP.NET 5 project. Either you execute them from the command line or your IDE like Visual Studio executes them for you. I like to understand what is going on behind the scenes so lets dive into that a little bit.</p><p>
<b>DNVM</b> - This is the .NET Version Manager. It is this utilities job to install the run times for your ASP.NET 5 app to use. You can have multiple runtime versions at the same time.</p><p>
<b>DNX</b> - Your ASP.NET 5 apps are launched via the DNX executable. DNX is the host for _all_ of your ASP.NET 5 applications - be it a console app or running inside of IIS. Even if it's transparent to you, behind the scenes DNX is running your application. If you remember this, life will be easier. DNX is the bootstrapper, compilation system, utilities, and more for running ASP.NET 5 on a variety of runtime versions like .NET 4.5.1, .NET 4.6, .NET Core, and Mono.</p><p>
If we look at the Startup.cs class generated in a File-&gt;New Project template we can see the following Configure method:</p><pre>// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
 loggerFactory.AddConsole(Configuration.GetSection("Logging"));
 loggerFactory.AddDebug();
 if (env.IsDevelopment())
 {
 app.UseBrowserLink();
 app.UseDeveloperExceptionPage();
 app.UseDatabaseErrorPage();
 }
 //..
}
</pre>
<p>
The <b>DNX</b> runtime actually is responsible for injecting IApplicationBuild, IHostingEnvironment, and ILoggerFactory into your application. You are communicating with DNX here. This is quite different than we've been used to with standard .NET applications.</p><p>
DNX does really has <a href="https://github.com/aspnet/Home/wiki/DNX-structure">four layers to its execution</a><br />
1. Boot the CLR for the version you have specified or default ie full .NET, CoreCLR etc.<br />
2. Create the LoaderContainer and IL Loaders (those resolve your assemblies)<br />
3. Resolve assemblies in project.json and call the entry point in your managed code<br />
4. Your application code executes</p><p>
As you can tell by now, DNX is the most important executable in this process.</p><p>
<b>DNU</b> - .NET Utilities. Yea - it's a generic name but think of this as your multi-tool to install packages (ie NuGet), build your app, or publish your app. There is no packages.config in your ASP.NET 5 projects, only the project.json.</p><p>
The <a href="https://github.com/aspnet/Home/wiki/Project.json-file">project.json file is the most important file in your project</a>. It defines the runtime version that will need to run your application. It also specifies dependencies for your applications - IE NuGet packages that will need to be restored before your project runs for the first time.</p><p>
Let's check out a portion of a project.json file with only the portions I want to discuss here shown below. The dependencies section are NuGet packages that your application depends on. Visual Studio will automatically restore these for you.</p><pre>{
 "dependencies": {
 "Microsoft.AspNet.Diagnostics": "1.0.0-rc1-final",
 "Microsoft.AspNet.IISPlatformHandler": "1.0.0-rc1-final",
 "Microsoft.AspNet.Mvc": "6.0.0-rc1-final",
 "Microsoft.AspNet.Mvc.TagHelpers": "6.0.0-rc1-final",
 "Microsoft.AspNet.Server.Kestrel": "1.0.0-rc1-final",
 "Microsoft.AspNet.StaticFiles": "1.0.0-rc1-final",
 "Microsoft.AspNet.Tooling.Razor": "1.0.0-rc1-final",
 "Microsoft.Extensions.Configuration.FileProviderExtensions" : "1.0.0-rc1-final",
 "Microsoft.Extensions.Configuration.Json": "1.0.0-rc1-final",
 "Microsoft.Extensions.Logging": "1.0.0-rc1-final",
 "Microsoft.Extensions.Logging.Console": "1.0.0-rc1-final",
 "Microsoft.Extensions.Logging.Debug": "1.0.0-rc1-final"
},
 "commands": {
 "web": "Microsoft.AspNet.Server.Kestrel"
 },
 "frameworks": {
 "dnx451":{},
 
 "dnxcore50": {}
 }
}</pre>
<p>Visual Studio Code will also prompt you to restore these packages.<a href="//1.bp.blogspot.com/-NCoBrST9lgs/VnfTML_3WGI/AAAAAAAABOk/VNHGpKxFHqA/s1600/VS%2BCode%2BRestore.png"></a>
<br />
Behind the scenes, both VS &amp; VS Code are simply using the command:</p><pre>dnu restore
</pre>
<p>
This command pulls in the packages from the NuGet source, which just pulls them down from the NuGet servers and caches them locally under for ex. c:\users\</p>\.dnx<p>
If you are using the command line (what both of the prior options use behind the scenes) you simply can run dnu restore in your project root folder.</p><div><p>
I want to stress that these commands will happen <i>automatically</i> in Visual Studio when you create/run your project. You only need to know these commands if you want to work at the command line, for ex. if you are working on Linux.</p><p>
Running your application happens via several methods. Publishing I'll cover in a future post. From Visual Studio simply click on the IIS Express button and it will have already configured IIS Express to support this via an <a href="https://github.com/aspnet/Home/wiki/Servers">HttpPlatformHandler</a>, which is a back-end module that launches processes that listen on HTTP Ports and proxies requests to them in IIS. This will make more sense in a second.</p></div>
<div>
<p><a href="//2.bp.blogspot.com/-UAHz6zOIDYI/VnfhNMzIF9I/AAAAAAAABP4/CqTZSDF8BoU/s1600/commands.png"></a></p><p>
See those other options above that say 'web' and 'ef'? Those are commands. <a href="http://docs.asp.net/en/latest/dnx/commands.html">Commands</a> are somemthing brand new to ASP.NET 5, they are simply a way to have multiple ways to load your application or perform actions on it. In the above example, we can run in IIS Express. If we click 'web' then this is going to launch Kestrel (still through DNX though) to act as a lighweight web server to run your app. If we want to generate code for Entity Framework to create our database for us we can use the 'ef' command to call into <a href="http://aspnetmvc.readthedocs.org/projects/mvc/en/latest/tutorials/mvc-with-entity-framework.html">Entity Framework's Code-First Migrations for ASP.NET 5</a>. BTW - if you have no idea about the power of EF Code-First Migrations, check out our free Microsoft Virtual Academy course on Entity Framework - specifically <a href="https://channel9.msdn.com/Series/Implementing-Entity-Framework-with-MVC/04">Module 4</a>.</p><p>
What is going on behind the scenes when all this happens? Its pretty simple.</p><pre>dnx web
</pre>
<p>
That's it :) If the command was named 'MyWebServer' instead of 'web' then of course it would be dnx mywebserver.</p></div>
<h3>
What is .NET Core?</h3>
<p>
.NET Core is a new lightweight open source .NET that is comprised of two components, CoreCLR which is essentially the runtime, and CoreFX which is the additional libraries like IO, Collections, etc that you would use in your applications. The major difference here is it is a pull mechanism. You start with a very lightweight framework and bring in the modules you need via NuGet packages. This is how the ASP.NET 5 templates are setup out of the box.<br />
<a href="//2.bp.blogspot.com/-xElafzePDBU/VnfrZ2YFMbI/AAAAAAAABQU/v8Kpni9Fo24/s1600/NETCORE.png"></a></p><p>
You can run ASP.NET 5 on the full .NET framework. There's baggage there though. Many extra libraries are loaded that you likely don't need.</p><p>
Looking back to the project.json file you can see the frameworks element. This will define what supported frameworks this app will run on. Here we see dnx451 (ie DNX support for the FULL .NET Framework 4.5.1) and dnxcore50 - the lightweight version.</p><p>
<a href="//2.bp.blogspot.com/-ebNcxvCoG7E/VnfzuJ3kNVI/AAAAAAAABQk/B9B79KzW5zg/s1600/frameworks.png"></a></p><p>
You have the ability to launch from Visual Studio and specify the default framework to load against by going into your project properties</p><p>
<a href="//1.bp.blogspot.com/-6AZha9X6vMw/Vnf1qNUvpbI/AAAAAAAABQw/nzxHw9XRNWU/s1600/app%2Bproperties.png"></a></p><p>
In reading up on .NET Core you'll find several ways of running this that are exclusive of each other. There's a dotnet utility if you want to compile or run a .NET Core app (the command line interface supports other operations as well) available at https://github.com/dotnet/cli. When it comes to ASP.NET 5 though, the .NET Core bits are loaded via the DNX utility. DNX allows you to specify which runtime you are launching the app with via:</p><pre>dnx --framework dnx451 web
</pre>
<p>
You have the flexibility to choose which framework you are going to run on. Marc Gravell from Stack Overflow fame has a good series on moving on over to DNX called <a href="http://blog.marcgravell.com/2015/11/the-road-to-dnx-part-1.html">The Road to DNX</a> as there are definite API changes if you've been used to developing on Full .NET (ex .NET 4.5) and moving to .NET Core.</p><h3>
Serving Up Content</h3><p>
ASP.NET 5 is decoupled from a web server implementation. DNX runs out of process and currently has support for IIS and the open source <a href="https://github.com/aspnet/KestrelHttpServer">Kestrel</a> web server. Previously the WebListener was an option, but that has been removed for RC as the move is towards using Kestrel to self-host. If you are running cross platform, the nginx web server/proxy can be used to proxy requests on over to Kestrel, just like the HttpPlatformHandler does for IIS. to be clear, IIS hosts DNX (through the HttpPlatformHandler), just as Kestrel hosts DNX as well. See the image below, note how in each case Kestrel is the host to listen to HTTP traffic on. The web server simply proxies traffic to it, whether its IIS or NGinx.</p><p>
<a href="//4.bp.blogspot.com/-t4NSxwFRsrs/VnhxcTfkTtI/AAAAAAAABRA/1J7gGQeEC_8/s1600/hosting%2Boptions.png"></a></p><p>
We talked about the project.json file which is a change from past versions of MVC, but it is also important to note that ASP.NET 5 projects will serve up static content by default only from the wwwroot folder. Views and Controllers work in much the same manner they did previously (if you are not familiar with that, I'll be posting a video shortly on this process).</p><h3>
Wrapping up</h3><p>
There's much more to go into, this was a high level overview of what differences ASP.NET 5 bring to the table. Stay tuned for some more posts on the lower level details, process execution / hosting, and more.</p><p>
Follow me <a href="https://twitter.com/adamtuliper/">Twitter to keep an eye on updates</a>, I try to post only useful stuff :)</p><p>
Enjoy!</p></div>||||I|||| Adam Tuliper's Development Tips

 Windows 10 Dev, Unity, ASP.NET, Entity Framework, Architecture, and various musings from a Microsoft Technical Evangelist

 Friday, October 19, 2018

 Woah. It's been a while.

 What a long time. I've been busy in the community, conferences, twitter, and now with lots of partner work I've been neglecting the blog a bit. Plus I have three young kids :) I've always enjoyed blogging and wow did that time go fast since my last post. Of course lots of fun things have been going on behind the scenes so I'm going to start surfacing some of those leanings here again. More posts to come shortly!!
 Posted by Adam Tuliper at 10:25 AM 5 comments:

 Thursday, June 2, 2016

 Getting Started with Game Development

 Game Dev - The Building Blocks

 At the Microsoft //build 2016 conference this year we created some great labs for the attendees to work on. Being a lover of all things game dev related, I worked on the Introduction to Unity lab. To prepare you for that hands on with real game artwork/code, I recommend you get started with the Building Blocks: Game Development course (below) from Microsoft Virtual Academy to understand the foundations of game development with Unity.



 This course covers topics from game objects & architecture to coding & animations and a bit of everything in between.

 If you aren't sure where to start with game dev, this is the perfect bite sized intro course for you and best of all, it is free. The course runs about two hours and covers 8 sections. Check out the module listing with this icon in the player below.




 Once you've finished up this course, be sure to check out the "Orks with Forks" lab we created which is the add-on content to this course. That video is at the bottom of this page.




 Build 2016 Code Labs (The add-on to the above course) can be watched below with the code open-sourced over here.


 Posted by Adam Tuliper at 12:51 AM 147 comments:

 Thursday, May 19, 2016

 Xamarin & Microsoft - //Build Reactions

 The.Time.Has.Come.



 One of the most exciting announcements at //Build (if not _the_ most exciting) is that the long-rumored joining of forces has finally happened between Microsoft and Xamarin. You can read the details here, but in a nutshell it goes like this:

 " Xamarin will be in every edition of Visual Studio, including the widely-available Visual Studio Community Edition, which is free for individual developers, open source projects, academic research, education, and small professional teams . Develop and publish native apps for iOS and Android with C# or F# from directly within Visual Studio with no limits on app size."

 This is huge. I've spoken with developers in the last couple years or so that really wanted to use Xamarin but shied away because of the costs. Now it is free. The Xamarin SDKs are also now open source, so you can view each platform's SDK code until your heart is content.

 Getting started is pretty simple but there are some requirements to be aware of. Building for iOS requires a Mac. Building for Android requires really...anything. Testing for either can be done on a PC with the new iOS Simulator James demos in the Channel 9 video below, which brings me to the next subject.

 I had the pleasure of talking with James Montemagno from Xamarin for my relatively new Channel 9 blog Adam's Garage (ps - hit me up with any content you want to see). You'll be hard pressed to find a more passionate mobile developer than James and he demos how easy it is to create a new app and a couple sample applications. Watch the video inline, or jump on over watch it on Channel 9 and be sure to read the notes below the video there to see the various specific subjects and direct links.




 Where to next?

 Get Visual Studio with Xamarin or install Xamarin into your existing Visual Studio instance. This url will allow you to do either. Once everything is setup, check out the following links for some great samples and further learning. Happy Coding :)

 * MyDriving Sample App - Azure IoT and Mobile Sample App
 * Check out the open source SDKs
 * Get some training at Xamarin University
 * Walkthrough - Creating Native UI apps with Xamaribn & Visual Studio
 * Creating a Xamarin Forms UWP app
 * Hello, iOS
 * Hello, Android
 * Xamarin Forms Quickstart
 Posted by Adam Tuliper at 2:23 AM 63 comments:

 Wednesday, April 13, 2016

 Getting Started with the HoloLens


 At Last

 The HoloLens team has announced shipments have started. Unboxing videos are popping up everywhere. My Twitter and Facebook feed are yielding random posts of friends opening theirs. Unity has finally announced the availability of the free Unity bits that supports the HoloLens. <EvilLaugh> The world is mine... at last </EvilLaugh>

 Want a HoloLens? You can apply for a dev kit and still get going with development this very minute.

 Alas, poor Adam, he has not a HoloLens and I'm assuming you don't either or you wouldn't be reading this - you'd be off exploring a galaxy or creating something in HoloStudio. But that's OK (for a short while anyways)! We don't need a HoloLens in our hands to be able to get started developing for one, just as you don't need a physical phone to start phone development. Yes, it's awesome to have one, yes the world is a better place when you deploy to a physical device - but we're developers. We can start developing now and not wait. Let's get to it!

 Don't want to read? Watch the video!




 Getting Started


 We just need a few things to get started......

 1. Windows 10 Pro, Enterprise, or Education
 2. Unity 5.4 HoloLens Preview (Install the Editor and the UWP Runtime - two separate installs)
 3. Visual Studio 2015 Update 2 (Be sure you check off UWP Tools 1.2+ and Windows 10 SDK if you use a custom install)
 4. HoloLens Emulator


 The above requirements are outlined here in more detail but I want to call one thing out. Your system must support Hyper-V or you can't run the emulator. Hyper-V is the virtualization system built into Windows, and is the reason you need Windows 10 Pro/Enterprise/Education and not Home Edition, as Home Edition doesn't support Hyper-V. If you are like me, you may run other software that uses Oracle VirtualBox (specifically Docker's Kitematic in my case) and I find myself switching Hyper-V on and off. If that fits you, there are various options to script out enabling/disabling Hyper-V.


 It's all so new

 That's true - it is all so new. This is brand new hot off the press API, bits, and documentation. As such you help the HoloLens folks and community out immensely by reporting anything in the HoloLens forums or asking any relevant questions there to help yourself out.

 To get started you have a handful of options

 * Create a new HoloLens app from the templates in Visual Studio (installed with the Emulator)
 * Create a new project in Unity and use for ex. the UnityEngine.VR.WSA namespace(s) to access HoloLens specific features
 * Use the samples to get started from the Holographic Academy here
 * Check out Galaxy Explorer
 * Pour through the HoloLens documentation


 There are a couple key concepts to be aware of. The first are the three pillars of Input on the HoloLens: Gaze, Gesture, and Voice. Gaze is what you are looking at (the device direction ie your head position, not your pupils). Gestures allow you to use your hand to interact with the Holograms and applications. Finally voice gives you access to the full Windows 10 speech engine.

 Next is the spatial duo. Spatial sound (shameless self promotion alert - check out our //Build talk on Spatial Audio in UWP to understand more about what spatial audio is) and spatial mapping. Spatial audio is literally making audio sound as close to real life positioning as possible. Imagine being able to hear a sound above you or behind you, even though you don't have any speakers above you or behind you (just on the side of your ears). That my friends, is the magic of spatial audio.

 Spatial Mapping is my favorite feature of the HoloLens which allows it to do some pretty neat things. Imagine dropping a hologram like a ball onto a real life table and it stays exactly where you dropped it. Now imagine you move the table fast out of the way and the ball now drops to the real floor and stops. This happens because the HoloLens is actually aware of its surroundings - it performs spatial mapping as shown in the image here. This works so well you can drop a hologram onto something, leave the room and come back later and it's still in the same place.




 It's all UWP

 We can create a UWP app from the templates in Visual Studio or we can create a new project in Unity and export to Visual Studio as a Direct3D or XAML UWP application. It's just a Visual Studio Solution, so we can open it up and deploy it to a device or an emulator.

 Keep in mind you can make near any Universal Windows Platform application run on the HoloLens. This is the power of UWP. You can run two different App Views on the HoloLens. In a 2D View, multiple apps from the Windows Store can run and you can pin them all around your environment (for ex to your walls). These would be 'flat' ie 2D apps. Secondly, you can run an app in a holographic view. The holographic view is when you have an app that will be drawing holograms. In this holographic view, no other app can run.

 To make UWP apps for the Hololens, as mentioned you can use Visual Studio and the code templates for the HoloLens, or Unity.


 But I Don't Know Unity!

 That OK. We have plenty of Unity learning resources available outside of the amazing content on Unity's learning site as well such as our video courses on Microsoft Virtual Academy

 * Build Blocks Game Development (**brand new quick start to go with GitHub lab below**)
 * Building Windows 10 Games with Unity 5
 * Developing 2D & 3D Games with Unity for Windows Jump Start


 and our recent GitHub based Unity Workshops from //Build are great for hour long quick starts

 * Introduction to Unity
 * Integrating Unity Games With Windows 10


 Easy Peasy

 Getting up and running is pretty straightforward. Next - let your creative juices flow and make some awesome experiences. One thing for you hard-core folks that love lower level information, read all about Hologram stability.

 Lastly, don't forget to tell the HoloLens Twitter Account about your cool projects! Happy Hacking :)


 Posted by Adam Tuliper at 7:21 AM 16 comments:

 Wednesday, December 16, 2015

 Understanding ASP.NET 5 and .NET Core


 **Update - New ASP.NET 5 video coming soon, follow me on Twitter to keep an eye on announcements **

 Most of my enterprise career was web based. In around 1997 my first major web project was to create the sign-up 3.5" floppy disks that were distributed with a good amount of Bell Atlantic phone books. This used all sorts of c-based cgi, Python scripts, and a mish mosh of tech. Things were so very very different then. We were actually the very first all Windows NT Internet Service Provider in the country. Good times but was web tech surely in its infancy.

 When ASP (Active Server Pages) first came out, it was amazing - it so we thought at the time as we spaghetti coded away our scripts. Fast forward nearly 20 years to ASP.NET 5. A technology that shares only letters with its predecessor. It's unbelievably fast. It runs an entirely different model than any of its predecessors. It is cross platform - even running on Linux and OSX. This is indeed a new Microsoft.


 What is ASP.NET 5?


 ASP.NET 5 is a new version of ASP.NET that has been built from the ground up with performance in mind, it is modular, and supports multiple platforms such as Windows, OSX, and Linux. ASP.NET 5 no longer contains Web Forms, it is purely MVC based and supports both C# and VB.NET. This may be a difficult point for some. Going forward, there was a lot of baggage Web Forms carried around and it was an abstraction to the web, which isn't how most modern frameworks work. As such, MVC 6 is the only supported option in ASP.NET 5. You can however still create Web Forms apps in Visual Studio 2015 (for ex. on .NET 4.6), just not for ASP.NET 5.

 The WebAPI technology (that released back with MVC 4) has now been merged into a unified controller model in ASP.NET 5. ASP.NET 5 now has out of the box support for Dependency Injection without the need for any third party DI containers (although several major ones work with it if that's your use case), and performance that can meet or exceed NodeJS applications. Let's not forget self hosting (running without installing into a web server) as well across all the supported platforms and of course you can host in IIS. You can be up and running self-hosting a RESTful api in minutes, publish to Azure and support millions of users, or host in your web server.

 ASP.NET 5 has a modular design so you only run the components that you need in your application. Prior versions of ASP.NET had significant overhead loading libraries that may have never been used. The memory footprint of ASP.NET 5 is significantly smaller and you can truly run your application now side by side with multiple versions of .NET and your application code.

 As stated in the docs website

 * New light-weight and modular HTTP request pipeline
 * Ability to host on IIS or self-host in your own process
 * Built on .NET Core, which supports true side-by-side app versioning
 * Ships entirely as NuGet packages
 * Integrated support for creating and using NuGet packages
 * Single aligned web stack for Web UI and Web APIs
 * Cloud-ready environment-based configuration
 * Built-in support for dependency injection
 * New tooling that simplifies modern web development
 * Build and run cross-platform ASP.NET apps on Windows, Mac and Linux
 * Open source and community focused
 ASP.NET 5 is now RC (Release Candidate) which means you will get support for it in production - although expect some further changes.


 What does ASP.NET 5 run on?


 ASP.NET 5 runs on Windows, OSX, Linux, and even Docker containers. It loads under .NET 4.5+, .NET 4.6, Mono, and .NET Core (more on that later).

 As a side note, there are some interesting future changes brewing here in determining what defines the .NET system your app will run on, and that is the .NET Platform Standard which defines a minimum set of base classes required to run your app.This is something to keep an eye on as it changes how our apps will refer to .NET to run.


 Getting Started

 First you need the bits for ASP.NET 5 which you can retrieve from http://get.asp.net. You can use all of your favorite code editors to create ASP.NET 5 applications. My preference is Visual Studio 2015 or Visual Studio Code as it has all the tooling to understand your apps better, but it's fairly trivial to do this with Sublime, Notepad, Brackets, etc.



 Next create an application. You have a couple options here. The first is in Visual Studio, do a File-New Project->Web, select ASP.NET Web Application. This launches then a new dialog where you could select the .NET 4.6.1 template for One ASP.NET (IE using MVC, Web API, and Web Forms all in one project) which I want to stress is not ASP.NET 5. You must select one of the ASP.NET 5 projects from the ASP.NET 5 section (makes sense, right?).






 Yeoman


 If you like using Yeoman (no idea what Yeoman is? Check out our free video here on Microsoft Virtual Academy) you can use Yeoman generator for ASP.NET 5. This will code-gen your base project for you. The commands (assuming you have NPM installed by installing NodeJS) are simply:

 npm install -g yo
 npm install -g generator-aspnet
 yo aspnet
 

 That will generate your project and you can simply right click on that generated folder and open in Visual Studio Code to see your entire project. There's actually a great article on this Visual Studio Code ASP.NET 5 workflow already, so I won't repeat that here.



 Project.json and the Three Commands

 There are three commands used in managing your ASP.NET 5 project. Either you execute them from the command line or your IDE like Visual Studio executes them for you. I like to understand what is going on behind the scenes so lets dive into that a little bit.

 DNVM - This is the .NET Version Manager. It is this utilities job to install the run times for your ASP.NET 5 app to use. You can have multiple runtime versions at the same time.

 DNX - Your ASP.NET 5 apps are launched via the DNX executable. DNX is the host for _all_ of your ASP.NET 5 applications - be it a console app or running inside of IIS. Even if it's transparent to you, behind the scenes DNX is running your application. If you remember this, life will be easier. DNX is the bootstrapper, compilation system, utilities, and more for running ASP.NET 5 on a variety of runtime versions like .NET 4.5.1, .NET 4.6, .NET Core, and Mono.

 If we look at the Startup.cs class generated in a File->New Project template we can see the following Configure method:

 // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
 public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
 {
 loggerFactory.AddConsole(Configuration.GetSection("Logging"));
 loggerFactory.AddDebug();
 
 if (env.IsDevelopment())
 {
 app.UseBrowserLink();
 app.UseDeveloperExceptionPage();
 app.UseDatabaseErrorPage();
 }
 //..
 }
 

 The DNX runtime actually is responsible for injecting IApplicationBuild, IHostingEnvironment, and ILoggerFactory into your application. You are communicating with DNX here. This is quite different than we've been used to with standard .NET applications.

 DNX does really has four layers to its execution
 1. Boot the CLR for the version you have specified or default ie full .NET, CoreCLR etc.
 2. Create the LoaderContainer and IL Loaders (those resolve your assemblies)
 3. Resolve assemblies in project.json and call the entry point in your managed code
 4. Your application code executes

 As you can tell by now, DNX is the most important executable in this process.

 DNU - .NET Utilities. Yea - it's a generic name but think of this as your multi-tool to install packages (ie NuGet), build your app, or publish your app. There is no packages.config in your ASP.NET 5 projects, only the project.json.

 The project.json file is the most important file in your project. It defines the runtime version that will need to run your application. It also specifies dependencies for your applications - IE NuGet packages that will need to be restored before your project runs for the first time.

 Let's check out a portion of a project.json file with only the portions I want to discuss here shown below. The dependencies section are NuGet packages that your application depends on. Visual Studio will automatically restore these for you.

 {
 "dependencies": {
 "Microsoft.AspNet.Diagnostics": "1.0.0-rc1-final",
 "Microsoft.AspNet.IISPlatformHandler": "1.0.0-rc1-final",
 "Microsoft.AspNet.Mvc": "6.0.0-rc1-final",
 "Microsoft.AspNet.Mvc.TagHelpers": "6.0.0-rc1-final",
 "Microsoft.AspNet.Server.Kestrel": "1.0.0-rc1-final",
 "Microsoft.AspNet.StaticFiles": "1.0.0-rc1-final",
 "Microsoft.AspNet.Tooling.Razor": "1.0.0-rc1-final",
 "Microsoft.Extensions.Configuration.FileProviderExtensions" : "1.0.0-rc1-final",
 "Microsoft.Extensions.Configuration.Json": "1.0.0-rc1-final",
 "Microsoft.Extensions.Logging": "1.0.0-rc1-final",
 "Microsoft.Extensions.Logging.Console": "1.0.0-rc1-final",
 "Microsoft.Extensions.Logging.Debug": "1.0.0-rc1-final"
 },
 
 "commands": {
 "web": "Microsoft.AspNet.Server.Kestrel"
 },
 
 "frameworks": {
 "dnx451":{},
 
 "dnxcore50": {}
 }
 }
 Visual Studio Code will also prompt you to restore these packages .
 Behind the scenes, both VS & VS Code are simply using the command:
 dnu restore
 

 This command pulls in the packages from the NuGet source, which just pulls them down from the NuGet servers and caches them locally under for ex. c:\users\\.dnx
 If you are using the command line (what both of the prior options use behind the scenes) you simply can run dnu restore in your project root folder.

 I want to stress that these commands will happen automatically in Visual Studio when you create/run your project. You only need to know these commands if you want to work at the command line, for ex. if you are working on Linux.


 Running your application happens via several methods. Publishing I'll cover in a future post. From Visual Studio simply click on the IIS Express button and it will have already configured IIS Express to support this via an HttpPlatformHandler, which is a back-end module that launches processes that listen on HTTP Ports and proxies requests to them in IIS. This will make more sense in a second.



 See those other options above that say 'web' and 'ef'? Those are commands. Commands are somemthing brand new to ASP.NET 5, they are simply a way to have multiple ways to load your application or perform actions on it. In the above example, we can run in IIS Express. If we click 'web' then this is going to launch Kestrel (still through DNX though) to act as a lighweight web server to run your app. If we want to generate code for Entity Framework to create our database for us we can use the 'ef' command to call into Entity Framework's Code-First Migrations for ASP.NET 5. BTW - if you have no idea about the power of EF Code-First Migrations, check out our free Microsoft Virtual Academy course on Entity Framework - specifically Module 4.

 What is going on behind the scenes when all this happens? Its pretty simple.

 dnx web
 

 That's it :) If the command was named 'MyWebServer' instead of 'web' then of course it would be dnx mywebserver.



 What is .NET Core?


 .NET Core is a new lightweight open source .NET that is comprised of two components, CoreCLR which is essentially the runtime, and CoreFX which is the additional libraries like IO, Collections, etc that you would use in your applications. The major difference here is it is a pull mechanism. You start with a very lightweight framework and bring in the modules you need via NuGet packages. This is how the ASP.NET 5 templates are setup out of the box.


 You can run ASP.NET 5 on the full .NET framework. There's baggage there though. Many extra libraries are loaded that you likely don't need.

 Looking back to the project.json file you can see the frameworks element. This will define what supported frameworks this app will run on. Here we see dnx451 (ie DNX support for the FULL .NET Framework 4.5.1) and dnxcore50 - the lightweight version.




 You have the ability to launch from Visual Studio and specify the default framework to load against by going into your project properties



 In reading up on .NET Core you'll find several ways of running this that are exclusive of each other. There's a dotnet utility if you want to compile or run a .NET Core app (the command line interface supports other operations as well) available at https://github.com/dotnet/cli. When it comes to ASP.NET 5 though, the .NET Core bits are loaded via the DNX utility. DNX allows you to specify which runtime you are launching the app with via:

 dnx --framework dnx451 web
 

 You have the flexibility to choose which framework you are going to run on. Marc Gravell from Stack Overflow fame has a good series on moving on over to DNX called The Road to DNX as there are definite API changes if you've been used to developing on Full .NET (ex .NET 4.5) and moving to .NET Core.



 Serving Up Content

 ASP.NET 5 is decoupled from a web server implementation. DNX runs out of process and currently has support for IIS and the open source Kestrel web server. Previously the WebListener was an option, but that has been removed for RC as the move is towards using Kestrel to self-host. If you are running cross platform, the nginx web server/proxy can be used to proxy requests on over to Kestrel, just like the HttpPlatformHandler does for IIS. to be clear, IIS hosts DNX (through the HttpPlatformHandler), just as Kestrel hosts DNX as well. See the image below, note how in each case Kestrel is the host to listen to HTTP traffic on. The web server simply proxies traffic to it, whether its IIS or NGinx.





 We talked about the project.json file which is a change from past versions of MVC, but it is also important to note that ASP.NET 5 projects will serve up static content by default only from the wwwroot folder. Views and Controllers work in much the same manner they did previously (if you are not familiar with that, I'll be posting a video shortly on this process).


 Wrapping up

 There's much more to go into, this was a high level overview of what differences ASP.NET 5 bring to the table. Stay tuned for some more posts on the lower level details, process execution / hosting, and more.

 Follow me Twitter to keep an eye on updates, I try to post only useful stuff :)

 Enjoy!

 Posted by Adam Tuliper at 1:20 PM 18 comments:

 Thursday, December 10, 2015

 Visual Studio Tools for Apache Cordova - A Developers Overview

 TL;DR; If you know JavaScript & HTML and want to write cross platform mobile apps that can utilize native device features (contacts, images, geolocation, etc) with a single codebase for iOS, Android, and Windows - Visual Studio Tools for Apache Cordova will enable you to do this. You can also remote debug from Visual Studio to iOS and on Android devices. We also have an Android Emulator for PC, a MacOS one coming out, and the ability to update your apps dynamically after users have already installed them - outside of the 'wait for the to approve updates' process.



 The nitty and the gritty

 There's some tooling available that will make your life way easier in developing cross platform mobile apps. But first, let's start in the beginning...


 What is Apache Cordova?

 Apache Cordova (previously PhoneGap - although that is now Adobe's productized version of Cordova) in short is an open source technology that allows you to create package an HTML/JavaScript application and allow JavaScript code to access native features (literally - any native feature) like contacts, battery level, accelerometer, images, and more on a device - again, from JavaScript.

 For example, to access the battery level on the phone you could simply use this code after you install the battery plugin (more on that below). This code would then work on Android, iOS, Windows Phone, BlackBerry, and more
 window.addEventListener("batterystatus", onBatteryStatus, false);
 
 function onBatteryStatus(info) {
 // Handle the online event
 console.log("Level: " + info.level + " isPlugged: " + info.isPlugged);
 }
 

 There are a lot of pieces you normally would have to install to get this all working. Like - 13 separate installs. The team at Microsoft has worked with the Apache Cordova team to ensure a seamless experience in Visual Studio.


 Where does Visual Studio fit in?

 Visual Studio is the development environment for nearly anything on the Microsoft platform. If you've been developing for a while, chances are you've heard of it if not used it. The thing is though - for many years it was just to integrate with Microsoft technology. That is no longer the case. You can build an Android app with it. You can even create an iOS app with Visual Studio using what we're going over here, or with the Xamarin support in Visual Studio, or even using Visual Studio to compile Objective-C for Windows. Today I'll show you how to create a cross platform app and debug it on Android. First you'll need some bits though.


 Installing the bits

 All you You need any Visual Studio 2013 or Visual Studio 2015 installation. Ideally though you want Visual Studio 2015 because the entire third party multi-install process is done for you. You can install Visual Studio Community for free here. During installation, ensure you check off the "Cross Platform Mobile Development" options. If you already have Visual Studio installed, you'll just simply need to run the setup again and select those options to add them on top of your installation. If you want to see the details of that, check out Install Visual Studio Tools for Apache Cordova




 Creating an app with a native feel

 One of the past criticisms of a Cordova/PhoneGap application is that it didn't 'feel' native. There are multiple ways around this perceived issue. Believe it or not, it is typically not a performance issue, but instead an issue of look and navigation. Android has a particular way it navigates between pages, iOS has another, Windows yet another, and they all have completely different looking icons in their apps. Take this example - which icons are from which OS? Clearly they have the same purpose but they show up differently. Differences as minor as this can be picked up by users.



 Our modern day advantage though is we can use hardware accelerated transitions and native looking icons easily inside of a Cordova application. Enter in the ionic framework and ionic icons. These two free projects from Ionic give your apps a native feel plus plenty of other features. Ionic has put together a template to use in Visual Studio that is a great starting point with Angular, multiple views, transistions, data binding, and of course an Apache Cordova template inside of Visual Studio. To use it, simply do File->New Project, click Online (on the left navigation) and search for the Ionic template as shown here and double click on it to install. The template install is a one time operation, you don't need to do this every project.




 Then you simply just create a new project with that template




 The first thing you'll want to do is ensure your project is updated to the latest Apache Cordova CLI (used to build your app) version by opening the config.xml. Visual Studio will give you a nice UI to edit this file, but you can just as easily right click on it and "Edit Code" to manually edit it as there are additional items in this file the UI doesn't show you (like iOS icon and splash screen information). If you don't have the latest version you may not see plugins available made using the latest version.




 I updated my version to 5.1.1 as that was the latest tested version as of this writing as noted here. Upon the next build of the project, the tooling will automatically grab the latest version and tel you about it in your build output

 ------ Installing Cordova tools cordova@5.1.1 for project from npm.
 


 Project Structure


 Once the project is created notice that the structure is pretty basic. The /merges folder contains files that will be added to your project only for a platform specific build - for example anything in /merges/android will be pulled into the project for an android build. If it exists in the /www folder it will be overwritten, otherwise just added to the project. The /plugins folder contains platform specific plugins (covered a tad more below), the /res folder contains platform specific resources, like Windows icons, iOS icons, and Android icons. Finally the /www folder contains all of your content like a website would, including the initial index.html which is the start page as noted in config.xml (which also contains things like the app name and description).




 Plugins

 If I want access to the battery information as in the prior JavaScript code, I need a plugin that someone wrote to do that. A plugin contains all the native code for each platform that is included during compilation for that platform. You can simply install the plugin and use the code posted at the beginning of the article. You can write your own plugins as well to do virtually anything on the native side from JavaScript. To add a plugin just open config.xml and find a plugin



 You can always see the installation and build details in the output window in Visual Studio so there's no mystery as to what is going on, as most of the installs use Node Package Manager (NPM).



 Once the plugin installs, note that there is native code for each and every supported platform. Here I've highlighted the iOS, Android, and Windows code that exists for the battery plugin. That code will be used during the compilation for any particular platform. Windows has already setup the Android tooling (and Windows of course) so when I run on a device, it is actually compiling that native code into the final build accessible by JavaScript. Plugins are cool - admit it :)





 Running and debugging

 You thankfully have many options when running your application. You can tell Visual Studio to deploy and run your app

 * on a Windows device
 * on an Android device
 * in the fast and free Android emulator that we made for PC
 * on iOS or iOS emulator via the Remote Agent for iOS
 * in the Ripple browser based emulator (I love this thing)

 I just want to take a moment to highlight our Android emulator. Geez - did I just say that? Yes - I write cross platform apps and games (and I own six PCs, a MacBook Pro, four Androids, three iPhones, and 8 Windows Phones) so need to test on various devices. Our Android emulator is fast. Visual Studio installs it (provided you checked it off during the install) and you can install APKs into it and test them. The tooling we're talking about today does it all automatically.

 There is a new fast Android Emulator coming out for the Mac as well for free. Your options today are to use Google's (which is purely software based and slow), or pay for a premium one. We are releasing a free one. You can find details of that on the Visual Studio Emulator for Android page.

 Ok - so I plugged in my Android Note 4, ensured my build was selected for Android (#1 below) and then clicked the Play button (#2 below) and the APK was auto-created, deployed to my phone, the app launched on my phone, and I was debugging it from Visual Studio and hitting breakpoints I setup in my code (just click any line of code on the left hand column and code execution will stop when it gets to that line). The same works for Windows of course, and the same works for running inside the emulator and also on iOS (via the Remote agent for iOS mentioned above).






 Here's a screenshot from my phone. Note that several images are missing. Which ones are they? Why are they missing? In this case my urls are incorrect and yielding a 404, Visual Studio will show you


 If I wanted to run this not on a device but instead in the Ripple emulator which runs in the browser, we can just choose one of the Ripple options (and Windows or iOS or Android on the left).



 Launching this now shows up in the Chrome. You'll note on the left and right hand side we can select various options like geolocation, forcing the battery level, etc. Hence, this is an emulator to make your app think a location is a certain area, you are running on a particular device, and more. Just keep in mind this isn't a virtual environment, it is a lightweight emulator.


 Updating applications after installation

 As with any software, the challenge comes into play when to update the app on the user's device. You have several solutions here depending on the type of app you have
 * Deploy a new version of the app to the store and wait for the user to update it
 * Wrap an existing website so content is served up live in your app
 * Have your own custom update process in place
 * Use Code Push to dynamically update application content in-place

 The first four are self explanatory. The last one though, Code Push, is a free service from Microsoft that is being developed specifically for this purpose. You use a couple command line operations and configure your project, and voila - you are ready for dynamic live updates on your deployed apps. You can find out more on the github project page.


 But I'm a modern web developer and use Gulp/Grunt/Bower/NPM!

 No problem! Visual Studio understands all of the above and you can implement any of those modern web technologies into the project and you can run your Gulp tasks, restore packages, etc. If you are unfamiliar with the above mentioned tools, check out my MSDN Magazine articles on Modern Web Development to get a quick start on them. In short though bower.json, package.json, gruntfile.js, and gulpfile.js are all understood by Visual Studio and can run your tasks (or not - your choice) and manage your package installations and restores inside your Visual Studio Tools for Apache Cordova application.


 MMMMMmmm Tacos.

 TACO, besides being the most delicious perfect food on the planet, is the Tools for Apache Cordova. This is a set of cross platform command line tools to generate Cordova apps. This is a separate code base than what was described above for Visual Studio and meant for those folks that love the command line. The following code would install TACO, create an app, and launch it a browser.

 #install taco command line interface
 npm install -g taco-cli
 #create the app
 taco create MyApp
 cd MyApp
 #launch in chrome (you could use ios or android instead of browser too!)
 taco platform add browser
 taco run browser
 



 Open Source

 The Apache Cordova project is open source. Visual Studio is not open source, but we do have Visual Studio Code, a cross-platform code editor that has recently been open sourced and includes extension support. Tooling is being worked on to be able to develop (including Intellisense) and debug your Cordova applications! You can get notified when this releases here and check out the video below.



 Wrapping up

 Visual Studio will generate your Windows executables, Windows Store packages, and also your Android APKs simply by using the Build menu in Visual Studio. Your iOS appplications need to be built on a Mac using a physical Mac or a service like Mac-In-Cloud, although I've simply installed the remote agent on my MacBook and did an iOS / Remote Device in the platform dropdowns and built/debugged from my MacBook to Visual Studio.




 What's Next


 Either create a new template from scratch or check out the Ionic template mentioned earlier or the SAP Fiori template, a pretty full featured app template that you can find here . There are so many additional things to write about that are useful. A common use case is to store data in a database from your apps. There's a great blog post about using Azure DocumentDB here from any JavaScript powered application.


 Visit the Visual Studio Tools for Apache Cordova team online - they do live sessions and Q/A!
 http://taco.visualstudio.com/live
 https://twitter.com/VSCordovaTools
 http://taco.visualstudio.com/en-us/docs/get-started-vs-tools-apache-cordova/

 Enjoy :)

 Posted by Adam Tuliper at 2:18 PM 61 comments:

 Wednesday, December 2, 2015

 Introduction to Visual Studio Dev Essentials - Dev Tools and Services for Everyone

 TL;DR; Enroll here to get free stuff to dev on any platform for any type of developer - all from one portal

 At Microsoft, I work with a lot of developer types. Cross Platform developers, web developers, app developers, game developers, and more. One of the challenges I see is in just getting the various tools and services a developer needs to do a task. They are never in one place. Take Apache Cordova. It's an awesome tech for cross platform development but there's like 12 different installs to get a complete tool set. This extends across every developer type. Enter in Visual Studio Dev Essentials.

 Visual Studio Dev Essentials is a program and portal that gives you access to software, services, free training from industry leaders on virtually any tech, and priority support.




 Who can benefit?

 If you touch any of these techs, then yes - you. It's free.
 * iOS
 * Android
 * Node.js
 * Python
 * Windows (Raspberry Pi, Phone, Desktop, Server, HoloLens, XBox, SurfaceHub, Laptop, Data)
 * IoT (Internet of Things)
 * Data Scientist - Machine Learning
 * Web (HTML, JavaScript, PHP, ASP.NET, ie all of you web devs)
 * Anyone doing cross platform development either writing native apps or Apache Cordova apps

 Free Software

 * Visual Studio Community Edition
 * Visual Studio Code
 * Windows VM for Windows Universal Development for 60 days
 * Parallels & Parallels Access - 90 day trial (only 14 on their site)

 PS - If you don't develop for Microsoft Tech, that's OK - read below for how this can help you on other tech. Our dev tools are some of the best on the planet for other tech too - like any web developer, Node.js, Python, Android, and cross platform development with Apache Cordova and Xamarin.


 Free Services

 * Visual Studio Team Services for five users (exclusive)
 * App Service Free Tier- Backend data and auth services for mobile and web (used by NBC, 3M and even used for the Super Bowl)
 * HockeyApp Free Tier (Haven't heard of this? Cross platform beta/feedback/crash reporting - more below though!)
 * Coming Soon - Azure $25 Credit (exclusive)
 * There are presently many free Azure services available too

 Free Learning

 * Pluralsight for three months (if you activated by Dec 31st its 6 months while supplies last)
 + PS ever hear of codeschool.com or digital-tutors.com? They are all part of Pluralsight
 * HackHands $25 credit with expert live help for coding via video conf and screenshare
 + Psps This is also a Pluralsight company :)
 * Xamarin University (coming very soon) - Classes and Certification for one of the best cross-platform technologies on the planet
 * WintellectNOW for three months
 * Priority support in MSDN Forums
 + No community reply after 48 hours? A Microsoft Engineer will address

 If you use Microsoft Technologies, you likely already know a lot of the above and are probably happy right now seeing it all on one place with extra free offerings. What about those that don't use our tech?


 Highlights for devs that don't use Microsoft Tech


 We have the IMHO best Android emulator around for Windows. Available as part of Visual Studio or a direct download. We use our own virtualization technology built into Windows to deliver performance that can be faster than being on an Android device. Python developers can benefit from Visual Studio to develop and remote debug Python scripts even on on MacOS & Linux.




 Microsoft or not - Cross Platform Developers









 HockeyApp - The name may be misleading. Hockey you say? Nope. Even better (and I like Hockey). This service provides cross platform beta distribution, crash logging, and live real user feedback capturing in your app for - get this - Android, iOS, OSX, Windows, Unity, and Xamarin. There's also mobile dev-ops and team management to round out an enticing service for near any app dev. You may have heard of TestFlight, which killed Android support over a year ago. No worries there's a solution for all platforms - try HockeyApp!

 Visual Studio Tools for Apache Cordova - I cannot stress how well this tech can work for cross platform app developers. If you've heard of PhoneGap.


 The Good Web Peoples


 Are you one of those old school but well learned web developers that don't like change or do you classify yourself as a new renegade developer that likes to change frameworks three times in a project because its new and compelling - or maybe brand new to web development? No problem either way. Visual Studio and Azure can help any of your projects get done faster and with fewer bugs. How so? Oh...

 * IDE integration with any JavaScript library
 * Integrated support for Grunt, Gulp, NPN, and Bower
 * Live JavaScript Debugging
 * Integrated GIT support
 * Live Cloud debugging on Azure (yes - a cloud deployed web app can be debugged locally)
 + Ps on a mildly related note, ever check out vorlon.js for remote JavaScript debugging?

 Azure also has cool features to deploy web apps like

 * Automatic deploy when you commit to GitHub
 * DropBox deployment
 * Data storage/retrieval from any web app with about three lines of code (including script reference!)
 * Multi platform push notifications (iOS, Android, Windows)
 * Integrate automatically with many providers like Twitter, Facebook, Office 365, SalesForce, Dynamics, OneDrive, Box, DropBox, Twilio, Facebook, Marketo, and more

 If using VS with Bower, NPM. Grunt, or Gulp piques your interested, check out my recent MSDN articles on using Bower, Grunt, and Gulp integrated right into Visual Studio
 Visual Studio - Bower: Modern Tools for Web Development

 Modern Tools for Web Development: Grunt and Gulp


 Go. For. It.

 Understand some of these services are available piecemeal. For example I can get Visual Studio 2015 Community by direct download. What's the catch then? Why use this? The advantage here is you have much more available - exclusive offers, but it also combines everything you need in one place so you don't have to go searching for the tools and services. It is a complete offering for your development.

 Hopefully something caught your eye as we really have something for near every developer type - even if Microsoft isn't your development platform (like using Visual Studio Code on OSX!).

 If you haven't yet - head right on over to enroll for free now.

 Any questions? Reach out to me @AdamTuliper



 Posted by Adam Tuliper at 10:45 AM 82 comments:
 Older Posts Home
 Subscribe to: Posts (Atom)


 StackOverflow

 Twitter Follow

 Follow @adamtuliper
 
 

 Followers

 Blogs I care about

 * Unity Technologies Blog
 Made with Unity Monthly: January 2023 roundup
 3 days ago
 * Jerry Nixon on Windows
 Waiting for Kestral
 3 months ago
 * The Sociable Geek
 Webstorm via the Command Line on Windows
 5 years ago
 
 
 Simple theme. Powered by Blogger.
ASP.NET 5 is a new version of ASP.NET that has been built from the ground up with performance in mind, it is modular, and supports multiple platforms such as Windows, OSX, and Linux. ASP.NET 5 no longer contains Web Forms, it is purely MVC based and supports both C# and VB.NET. This may be a difficult point for some. Going forward, there was a lot of baggage Web Forms carried around and it was an abstraction to the web, which isn't how most modern frameworks work. As such, MVC 6 is the only supported option in ASP.NET 5. You can however still create Web Forms apps in Visual Studio 2015 (for ex. on .NET 4.6), just not for ASP.NET 5. 
The WebAPI technology (that released back with MVC 4) has now been merged into a unified controller model in ASP.NET 5. ASP.NET 5 now has out of the box support for Dependency Injection without the  need for any third party DI containers (although several major ones work with it if that's your use case), and performance that can meet or exceed NodeJS applications. Let's not forget self hosting (running without installing into a web server) as well across all the supported platforms and of course you can host in IIS. You can be up and running self-hosting a RESTful api in minutes, publish to Azure and support millions of users, or host in your web server. 
ASP.NET 5 has a modular design so you only run the components that you need in your application. Prior versions of ASP.NET had significant overhead loading libraries that may have never been used. The memory footprint of ASP.NET 5 is significantly smaller and you can truly run your application now side by side with multiple versions of .NET and your application code. 
As stated in the docs website 
New light-weight and modular HTTP request pipeline
Ability to host on IIS or self-host in your own process
Built on .NET Core, which supports true side-by-side app versioning
Ships entirely as NuGet packages
Integrated support for creating and using NuGet packages
Single aligned web stack for Web UI and Web APIs
Cloud-ready environment-based configuration
Built-in support for dependency injection
New tooling that simplifies modern web development
Build and run cross-platform ASP.NET apps on Windows, Mac and Linux
Open source and community focused
ASP.NET 5 is now RC (Release Candidate) which means you will get support for it in production - although expect some further changes.
 
What does ASP.NET 5 run on?
ASP.NET 5  runs on  Windows, OSX, Linux, and even Docker containers. It  loads under . NET 4.5+, .NET 4.6, Mono, and .NET Core (more on that later). 
As a side note, there are some interesting future changes brewing here in determining what defines the .NET system your app will run on, and that is the .NET Platform Standard which defines a minimum set of base classes required to run your app.This is something to keep an eye on as it changes how our apps will refer to .NET to run. 
Getting Started 
First you need the bits for ASP.NET 5 which you can retrieve from http://get.asp.net. You can use all of your favorite code editors to create ASP.NET 5 applications. My preference is Visual Studio 2015 or Visual Studio Code as it has all the tooling to understand your apps better, but it's fairly trivial to do this with Sublime, Notepad, Brackets, etc. 
   
Next create an application. You have a couple options here. The first is in Visual Studio, do a File-New Project-&gt;Web, select ASP.NET Web Application. This launches then a new dialog where you could select the .NET 4.6.1 template for One ASP.NET (IE using MVC, Web API, and Web Forms all in one project) which I want to stress is not ASP.NET 5. You must select one of the ASP.NET 5 projects from the ASP.NET 5 section (makes sense, right?). 
   
   
Yeoman
   
If you like using Yeoman (no idea what Yeoman is? Check out our free video here on Microsoft Virtual Academy) you can use Yeoman generator for ASP.NET 5. This will code-gen your base project for you. The commands (assuming you have NPM installed by installing NodeJS) are simply: npm install -g yo
npm install -g generator-aspnet
yo aspnet
That will generate your project and you can simply right click on that generated folder and open in Visual Studio Code to see your entire project. There's actually a great article on this Visual Studio Code ASP.NET 5 workflow already, so I won't repeat that here.
   
Project.json and the Three Commands 
There are three commands used in managing your ASP.NET 5 project. Either you execute them from the command line or your IDE like Visual Studio executes them for you. I like to understand what is going on behind the scenes so lets dive into that a little bit. 
DNVM - This is the .NET Version Manager. It is this utilities job to install the run times for your ASP.NET 5 app to use. You can have multiple runtime versions at the same time. 
DNX - Your ASP.NET 5 apps are launched via the DNX executable. DNX is the host for _all_ of your ASP.NET 5 applications - be it a console app or running inside of IIS. Even if it's transparent to you, behind the scenes DNX is running your application. If you remember this, life will be easier. DNX is the bootstrapper, compilation system, utilities, and more for running ASP.NET 5 on a variety of runtime versions like .NET 4.5.1, .NET 4.6, .NET Core, and Mono. 
If we look at the Startup.cs class generated in a File-&gt;New Project template we can see the following Configure method: // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
 loggerFactory.AddConsole(Configuration.GetSection("Logging"));
 loggerFactory.AddDebug();
 if (env.IsDevelopment())
 {
 app.UseBrowserLink();
 app.UseDeveloperExceptionPage();
 app.UseDatabaseErrorPage();
 }
 //..
}
The DNX runtime actually is responsible for injecting IApplicationBuild, IHostingEnvironment, and ILoggerFactory into your application. You are communicating with DNX here. This is quite different than we've been used to with standard .NET applications. 
DNX does really has four layers to its execution 
1. Boot the CLR for the version you have specified or default ie full .NET, CoreCLR etc.
2. Create the LoaderContainer and IL Loaders (those resolve your assemblies)
3. Resolve assemblies in project.json and call the entry point in your managed code
4. Your application code executes 
As you can tell by now, DNX is the most important executable in this process. 
DNU - .NET Utilities. Yea - it's a generic name but think of this as your multi-tool to install packages (ie NuGet), build your app, or publish your app. There is no packages.config in your ASP.NET 5 projects, only the project.json. 
The project.json file is the most important file in your project. It defines the runtime version that will need to run your application. It also specifies dependencies for your applications - IE NuGet packages that will need to be restored before your project runs for the first time. 
Let's check out a portion of a project.json file with only the portions I want to discuss here shown below. The dependencies section are NuGet packages that your application depends on. Visual Studio will automatically restore these for you. {
 "dependencies": {
 "Microsoft.AspNet.Diagnostics": "1.0.0-rc1-final",
 "Microsoft.AspNet.IISPlatformHandler": "1.0.0-rc1-final",
 "Microsoft.AspNet.Mvc": "6.0.0-rc1-final",
 "Microsoft.AspNet.Mvc.TagHelpers": "6.0.0-rc1-final",
 "Microsoft.AspNet.Server.Kestrel": "1.0.0-rc1-final",
 "Microsoft.AspNet.StaticFiles": "1.0.0-rc1-final",
 "Microsoft.AspNet.Tooling.Razor": "1.0.0-rc1-final",
 "Microsoft.Extensions.Configuration.FileProviderExtensions" : "1.0.0-rc1-final",
 "Microsoft.Extensions.Configuration.Json": "1.0.0-rc1-final",
 "Microsoft.Extensions.Logging": "1.0.0-rc1-final",
 "Microsoft.Extensions.Logging.Console": "1.0.0-rc1-final",
 "Microsoft.Extensions.Logging.Debug": "1.0.0-rc1-final"
},
 "commands": {
 "web": "Microsoft.AspNet.Server.Kestrel"
 },
 "frameworks": {
 "dnx451":{},
 
 "dnxcore50": {}
 }
}
Visual Studio Code will also prompt you to restore these packages. 
Behind the scenes, both VS &amp; VS Code are simply using the command: dnu restore
This command pulls in the packages from the NuGet source, which just pulls them down from the NuGet servers and caches them locally under for ex. c:\users\\.dnx
If you are using the command line (what both of the prior options use behind the scenes) you simply can run dnu restore in your project root folder.  
I want to stress that these commands will happen automatically in Visual Studio when you create/run your project. You only need to know these commands if you want to work at the command line, for ex. if you are working on Linux. 
Running your application happens via several methods. Publishing I'll cover in a future post. From Visual Studio simply click on the IIS Express button and it will have already configured IIS Express to support this via an HttpPlatformHandler, which is a back-end module that launches processes that listen on HTTP Ports and proxies requests to them in IIS. This will make more sense in a second. 
    
See those other options above that say 'web' and 'ef'? Those are commands. Commands are somemthing brand new to ASP.NET 5, they are simply a way to have multiple ways to load your application or perform actions on it. In the above example, we can run in IIS Express. If we click 'web' then this is going to launch Kestrel (still through DNX though) to act as a lighweight web server to run your app. If we want to generate code for Entity Framework to create our database for us we can use the 'ef' command to call into Entity Framework's Code-First Migrations for ASP.NET 5. BTW - if you have no idea about the power of EF Code-First Migrations, check out our free Microsoft Virtual Academy course on Entity Framework - specifically Module 4. 
What is going on behind the scenes when all this happens? Its pretty simple. dnx web
That's it :) If the command was named 'MyWebServer' instead of 'web' then of course it would be dnx mywebserver. 
What is .NET Core?
.NET Core is a new lightweight open source .NET that is comprised of two components, CoreCLR which is essentially the runtime, and CoreFX which is the additional libraries like IO, Collections, etc that you would use in your applications. The major difference here is it is a pull mechanism. You start with a very lightweight framework and bring in the modules you need via NuGet packages. This is how the ASP.NET 5 templates are setup out of the box.
   
You can run ASP.NET 5 on the full .NET framework. There's baggage there though. Many extra libraries are loaded that you likely don't need. 
Looking back to the project.json file you can see the frameworks element. This will define what supported frameworks this app will run on. Here we see dnx451 (ie DNX support for the FULL .NET Framework 4.5.1) and dnxcore50 - the lightweight version. 
   
You have the ability to launch from Visual Studio and specify the default framework to load against by going into your project properties 
   
In reading up on .NET Core you'll find several ways of running this that are exclusive of each other. There's a dotnet utility if you want to compile or run a .NET Core app (the command line interface supports other operations as well) available at https://github.com/dotnet/cli. When it comes to ASP.NET 5 though, the .NET Core bits are loaded via the DNX utility. DNX allows you to specify which runtime you are launching the app with via: dnx --framework dnx451 web
You have the flexibility to choose which framework you are going to run on. Marc Gravell from Stack Overflow fame has a good series on moving on over to DNX called The Road to DNX as there are definite API changes if you've been used to developing on Full .NET (ex .NET 4.5) and moving to .NET Core. 
Serving Up Content 
ASP.NET 5 is decoupled from a web server implementation. DNX runs out of process and currently has support for IIS and the open source Kestrel web server. Previously the WebListener was an option, but that has been removed for RC as the move is towards using Kestrel to self-host. If you are running cross platform, the nginx web server/proxy can be used to proxy requests on over to Kestrel, just like the HttpPlatformHandler does for IIS. to be clear, IIS hosts DNX (through the HttpPlatformHandler), just as Kestrel hosts DNX as well. See the image below, note how in each case Kestrel is the host to listen to HTTP traffic on. The web server simply proxies traffic to it, whether its IIS or NGinx. 
   
We talked about the project.json file which is a change from past versions of MVC, but it is also important to note that ASP.NET 5 projects will serve up static content by default only from the wwwroot folder. Views and Controllers work in much the same manner they did previously (if you are not familiar with that, I'll be posting a video shortly on this process). 
Wrapping up 
There's much more to go into, this was a high level overview of what differences ASP.NET 5 bring to the table. Stay tuned for some more posts on the lower level details, process execution / hosting, and more. 
Follow me Twitter to keep an eye on updates, I try to post only useful stuff :) 
Enjoy!
Adam Tuliper's Development Tips
http://www.adamtuliper.com/
https://addyosmani.com/blog/
Addy Osmani
2022-12-16
Addy Osmani
<div><section>
 
 <header>
 
 <a href="/press">
 
 </a>
 
 
 <h2>Addy Osmani</h2>
 <h2>Engineering Manager at Google working on Chrome</h2>
 
 </header>
 
 </section>||||I||||Home GitHub Press Biography LinkedIn Twitter Subscribe Shop Blog

 Addy Osmani

 Engineering Manager at Google working on Chrome

 * Dec 16 2022
 Prioritizing tasks using the Eisenhower matrix

 * Dec 15 2022
 Running inclusive meetings

 * Dec 10 2022
 Effective Learning with Bloom's Taxonomy

 * Dec 04 2022
 First Principles for Software Engineers

 * Nov 30 2022
 A coach, a mentor and a sponsor

 * Nov 28 2022
 Debugging Tactics

 * Nov 17 2022
 Focus on high-leverage activities.

 * Nov 12 2022
 Debugging teams with the Lencioni Model

 * Nov 07 2022
 Managing Complex Change

 * Aug 16 2022
 Communicating Effectively As A Tech Lead

Older Posts →

© Copyright 2023 Addy Osmani

Home GitHub Press Biography LinkedIn Twitter Subscribe Shop Blog
 Addy Osmani
 Engineering Manager at Google working on Chrome
 * Dec 16 2022
 Prioritizing tasks using the Eisenhower matrix
 * Dec 15 2022
 Running inclusive meetings
 * Dec 10 2022
 Effective Learning with Bloom's Taxonomy
 * Dec 04 2022
 First Principles for Software Engineers
 * Nov 30 2022
 A coach, a mentor and a sponsor
 * Nov 28 2022
 Debugging Tactics
 * Nov 17 2022
 Focus on high-leverage activities.
 * Nov 12 2022
 Debugging teams with the Lencioni Model
 * Nov 07 2022
 Managing Complex Change
 * Aug 16 2022
 Communicating Effectively As A Tech Lead
Older Posts →
© Copyright 2023 Addy Osmani
Addy Osmani
https://addyosmani.com/blog/
https://danluu.com/
None
1999-01-01
None
<div><div><div><p>xx/xx</p><p><a href="#patreon">Patreon posts</a></p></div><div><p>09/22</p><p><a href="https://danluu.com/futurist-predictions/">Futurist prediction methods and accuracy</a></p></div><div><p>04/22</p><p><a href="https://danluu.com/simple-architectures/">In defense of simple architectures</a></p></div><div><p>03/22</p><p><a href="https://danluu.com/nothing-works/">Why is it so hard to buy things that work well?</a></p></div><div><p>02/22</p><p><a href="https://danluu.com/talent/">Misidentifying talent</a></p></div><div><p>02/22</p><p><a href="https://danluu.com/cache-incidents/">A decade of major cache incidents at Twitter</a></p></div><div><p>02/22</p><p><a href="https://danluu.com/cocktail-ideas/">Cocktail party ideas</a></p></div><div><p>12/21</p><p><a href="https://danluu.com/cgroup-throttling/">The container throttling problem</a></p></div><div><p>12/21</p><p><a href="https://danluu.com/writing-non-advice/">Some thoughts on writing</a></p></div><div><p>12/21</p><p><a href="https://danluu.com/latency-pitfalls/">Some latency measurement pitfalls</a></p></div><div><p>11/21</p><p><a href="https://danluu.com/corrections/">Major errors on this blog (and their corrections)</a></p></div><div><p>11/21</p><p><a href="https://danluu.com/people-matter/">Individuals matter</a></p></div><div><p>11/21</p><p><a href="https://danluu.com/culture/">Culture matters</a></p></div><div><p>10/21</p><p><a href="https://danluu.com/look-stupid/">Willingness to look stupid</a></p></div><div><p>10/21</p><p><a href="https://danluu.com/learn-what/">What to learn</a></p></div><div><p>10/21</p><p><a href="https://danluu.com/productivity-velocity/">Some reasons to work on productivity and velocity</a></p></div><div><p>09/21</p><p><a href="https://danluu.com/in-house/">The value of in-house expertise</a></p></div><div><p>08/21</p><p><a href="https://danluu.com/why-benchmark/">Some reasons to measure</a></p></div><div><p>12/20</p><p><a href="https://danluu.com/essential-complexity/">Against essential and accidental complexity</a></p></div><div><p>06/20</p><p><a href="https://danluu.com/car-safety/">How do cars fare in crash tests they're not specifically optimized for?</a></p></div><div><p>05/20</p><p><a href="https://danluu.com/tracing-analytics/">A simple way to get more value from tracing</a></p></div><div><p>05/20</p><p><a href="https://danluu.com/metrics-analytics/">A simple way to get more value from metrics</a></p></div><div><p>03/20</p><p><a href="https://danluu.com/corp-eng-blogs/">How (some) good corporate engineering blogs are written</a></p></div><div><p>03/20</p><p><a href="https://danluu.com/cli-complexity/">The growth of command line options, 1979-Present</a></p></div><div><p>02/20</p><p><a href="https://danluu.com/discontinuities/">Suspicious discontinuities</a></p></div><div><p>02/20</p><p><a href="https://danluu.com/p95-skill/">95%-ile isn't that good</a></p></div><div><p>01/20</p><p><a href="https://danluu.com/algorithms-interviews/">Algorithms interviews: theory vs. practice</a></p></div><div><p>07/19</p><p><a href="https://danluu.com/deconstruct-files/">Files are fraught with peril</a></p></div><div><p>02/19</p><p><a href="https://danluu.com/overwatch-gender/">Randomized trial on gender in Overwatch</a></p></div><div><p>12/17</p><p><a href="https://danluu.com/input-lag/">Computer latency: 1977-2017</a></p></div><div><p>11/17</p><p><a href="https://danluu.com/bad-decisions/">How good are decisions?</a></p></div><div><p>11/17</p><p><a href="https://danluu.com/android-updates/">How out of date are Android devices?</a></p></div><div><p>11/17</p><p><a href="https://danluu.com/ui-compatibility/">UI backwards compatibility</a></p></div><div><p>10/17</p><p><a href="https://danluu.com/filesystem-errors/">Filesystem error handling</a></p></div><div><p>10/17</p><p><a href="https://danluu.com/keyboard-latency/">Keyboard latency</a></p></div><div><p>08/17</p><p><a href="https://danluu.com/branch-prediction/">Branch prediction</a></p></div><div><p>08/17</p><p><a href="https://danluu.com/sattolo/">Sattolo's algorithm</a></p></div><div><p>07/17</p><p><a href="https://danluu.com/term-latency/">Terminal latency</a></p></div><div><p>06/17</p><p><a href="https://danluu.com/keyboard-v-mouse/">Keyboard v. mouse</a></p></div><div><p>06/17</p><p><a href="https://danluu.com/startup-options/">Startup options v. cash</a></p></div><div><p>02/17</p><p><a href="https://danluu.com/web-bloat/">The modern web on a slow connection</a></p></div><div><p>10/16</p><p><a href="https://danluu.com/hn-comments/">HN: the good parts</a></p></div><div><p>10/16</p><p><a href="https://danluu.com/programming-books/">Programming book recommendations and anti-recommendations</a></p></div><div><p>10/16</p><p><a href="https://danluu.com/hiring-lemons/">Hiring and the market for lemons</a></p></div><div><p>10/16</p><p><a href="https://danluu.com/sounds-easy/">I could do that in a weekend!</a></p></div><div><p>09/16</p><p><a href="https://danluu.com/bimodal-compensation/">Is dev compensation bimodal?</a></p></div><div><p>09/16</p><p><a href="https://danluu.com/learning-to-program/">How I learned to program</a></p></div><div><p>08/16</p><p><a href="https://danluu.com/concurrency-bugs/">Notes on concurrency bugs</a></p></div><div><p>04/16</p><p><a href="https://danluu.com/programming-blogs/">Some programming blogs to consider reading</a></p></div><div><p>04/16</p><p><a href="https://danluu.com/google-sre-book/">Google SRE book</a></p></div><div><p>03/16</p><p><a href="https://danluu.com/programmer-moneyball/">We only hire the trendiest</a></p></div><div><p>01/16</p><p><a href="https://danluu.com/perf-tracing/">Sampling v. tracing</a></p></div><div><p>01/16</p><p><a href="https://danluu.com/cpu-bugs/">CPU bugs</a></p></div><div><p>12/15</p><p><a href="https://danluu.com/wat/">Normalization of deviance</a></p></div><div><p>12/15</p><p><a href="https://danluu.com/startup-tradeoffs/">Big companies v. startups</a></p></div><div><p>12/15</p><p><a href="https://danluu.com/file-consistency/">Files are hard</a></p></div><div><p>11/15</p><p><a href="https://danluu.com/why-ecc/">Why use ECC?</a></p></div><div><p>11/15</p><p><a href="https://danluu.com/butler-lampson-1999/">What's worked in Computer Science: 1999 v. 2015</a></p></div><div><p>11/15</p><p><a href="https://danluu.com/infinite-disk/">Infinite disk</a></p></div><div><p>10/15</p><p><a href="https://danluu.com/intel-cat/">Why Intel added cache partitioning</a></p></div><div><p>09/15</p><p><a href="https://danluu.com/limplock/">Slowlock</a></p></div><div><p>08/15</p><p><a href="https://danluu.com/yegge-predictions/">Steve Yegge's prediction record</a></p></div><div><p>08/15</p><p><a href="https://danluu.com/postmortem-lessons/">Reading postmortems</a></p></div><div><p>05/15</p><p><a href="https://danluu.com/slashdot-sourceforge/">Slashdot and Sourceforge</a></p></div><div><p>05/15</p><p><a href="https://danluu.com/googlebot-monopoly/">The googlebot monopoly</a></p></div><div><p>05/15</p><p><a href="https://danluu.com/boring-languages/">A defense of boring languages</a></p></div><div><p>05/15</p><p><a href="https://danluu.com/monorepo/">Advantages of monorepos</a></p></div><div><p>03/15</p><p><a href="https://danluu.com/dunning-kruger/">Reading citations is easier than most people think</a></p></div><div><p>03/15</p><p><a href="https://danluu.com/testing/">Given that devs spend the effort they do on testing, what can we do to improve testing?</a></p></div><div><p>03/15</p><p><a href="https://danluu.com/navigate-url/">What happens when you load a URL?</a></p></div><div><p>03/15</p><p><a href="https://danluu.com/percentile-latency/">Goodhearting IQ, cholesterol, and tail latency</a></p></div><div><p>02/15</p><p><a href="https://danluu.com/customer-service/">AI doesn't have to be that good</a></p></div><div><p>02/15</p><p><a href="https://danluu.com/cpu-backdoors/">CPU backdoors</a></p></div><div><p>01/15</p><p><a href="https://danluu.com/blog-ads/">Blog monetization</a></p></div><div><p>01/15</p><p><a href="https://danluu.com/new-cpu-features/">What's new in CPUs since the 80s?</a></p></div><div><p>12/14</p><p><a href="https://danluu.com/julialang/">A review of the Julia language</a></p></div><div><p>12/14</p><p><a href="https://danluu.com/integer-overflow/">Integer overflow checking cost</a></p></div><div><p>12/14</p><p><a href="https://danluu.com/malloc-tutorial/">Malloc tutorial</a></p></div><div><p>12/14</p><p><a href="https://danluu.com/tech-discrimination/">Markets, discrimination, and "lowering the bar"</a></p></div><div><p>11/14</p><p><a href="https://danluu.com/linux-devs-say/">TF-IDF linux commits</a></p></div><div><p>11/14</p><p><a href="https://danluu.com/everything-is-broken/">One week of bugs</a></p></div><div><p>11/14</p><p><a href="https://danluu.com/octopress-speedup/">Speeding up this site by 50x</a></p></div><div><p>11/14</p><p><a href="https://danluu.com/broken-builds/">Build uptime</a></p></div><div><p>11/14</p><p><a href="https://danluu.com/empirical-pl/">Literature review on the benefits of static types</a></p></div><div><p>11/14</p><p><a href="https://danluu.com/clwb-pcommit/">CLWB and PCOMMIT</a></p></div><div><p>11/14</p><p><a href="https://danluu.com/2choices-eviction/">Caches: LRU v. random</a></p></div><div><p>11/14</p><p><a href="https://danluu.com/tests-v-reason/">Testing v. informal reasoning</a></p></div><div><p>10/14</p><p><a href="https://danluu.com/assembly-intrinsics/">Assembly v. intrinsics</a></p></div><div><p>04/14</p><p><a href="https://danluu.com/bugalytics/">Bug analytics</a></p></div><div><p>03/14</p><p><a href="https://danluu.com/edit-binary/">Editing binaries</a></p></div><div><p>03/14</p><p><a href="https://danluu.com/gender-gap/">That bogus gender gap article</a></p></div><div><p>03/14</p><p><a href="https://danluu.com/anon-benchmark/">A solution to DeWitt clauses</a></p></div><div><p>02/14</p><p><a href="https://danluu.com/teach-debugging/">Why don't schools teach debugging?</a></p></div><div><p>01/14</p><p><a href="https://danluu.com/math-bias/">Do programmers need math?</a></p></div><div><p>01/14</p><p><a href="https://danluu.com/3c-conflict/">Data alignment and caches</a></p></div><div><p>12/13</p><p><a href="https://danluu.com/linear-hammer/">PCA is not a panacea</a></p></div><div><p>11/13</p><p><a href="https://danluu.com/hardware-unforgiving/">The complexity of knowledge and skill transfer</a></p></div><div><p>10/13</p><p><a href="https://danluu.com/discourage-oss/">How to discourage open source contributions</a></p></div><div><p>10/13</p><p><a href="https://danluu.com/randomize-hn/">Randomize HN</a></p></div><div><p>09/13</p><p><a href="https://danluu.com/pl-troll/">Writing safe Verilog</a></p></div><div><p>09/13</p><p><a href="https://danluu.com/why-hardware-development-is-hard/">Verilog is weird</a></p></div><p> ↑ Public posts </p><hr /><p> ↓Patreon posts</p><div><p>02/22</p><p><a href="https://www.patreon.com/posts/62933244">Pop analogies</a></p></div><div><p>02/22</p><p><a href="https://www.patreon.com/posts/62223485">Ironic, self-refuting, rebuttals</a></p></div><div><p>01/22</p><p><a href="https://www.patreon.com/posts/61946482">Doing good engineering with bad mental models</a></p></div><div><p>01/22</p><p><a href="https://www.patreon.com/posts/61804963">The backlash against doing work (no, not that one)</a></p></div><div><p>12/21</p><p><a href="https://www.patreon.com/posts/60185075">My style vs. monetization</a></p></div><div><p>12/21</p><p><a href="https://www.patreon.com/posts/59812602">Phrases that mean their opposite</a></p></div><div><p>11/21</p><p><a href="https://www.patreon.com/posts/59116025">Cost of living adjustments and location-based pay</a></p></div><div><p>11/21</p><p><a href="https://www.patreon.com/posts/58713950">Writing down the most obvious things</a></p></div><div><p>08/21</p><p><a href="https://www.patreon.com/posts/54329188">On Alan Kay's claim that computers would be 1000x faster if we listened to him</a></p></div><div><p>07/21</p><p><a href="https://www.patreon.com/posts/54154288">Some reasons to measure (draft)</a></p></div><div><p>03/21</p><p><a href="https://www.patreon.com/posts/48172059">What to learn? (draft)</a></p></div><div><p>01/21</p><p><a href="https://www.patreon.com/posts/46629220">The glorification of elite teams</a></p></div><div><p>09/20</p><p><a href="https://www.patreon.com/posts/41737660">What does it mean that some Google execs think they gave away the farm with k8s?</a></p></div><div><p>07/20</p><p><a href="https://www.patreon.com/posts/38866741">You'd have to be very smart to come up with that</a></p></div><div><p>06/20</p><p><a href="https://www.patreon.com/posts/38563254">Reactions to the NYT's potential doxxing of Scott Alexander</a></p></div><div><p>06/20</p><p><a href="https://www.patreon.com/posts/38148848">Why are you still working?</a></p></div><div><p>05/20</p><p><a href="https://www.patreon.com/posts/36719138">Blog posts I'm not publishing</a></p></div><div><p>02/20</p><p><a href="https://www.patreon.com/posts/33452460">Another case against taking startup employee equity</a></p></div><div><p>12/19</p><p><a href="https://www.patreon.com/posts/32211481">Working at an ad supported company (or not)</a></p></div><div><p>11/19</p><p><a href="https://www.patreon.com/posts/31646816">Startup vs. big company compensation in practice</a></p></div><div><p>09/19</p><p><a href="https://www.patreon.com/posts/29753680">We're just like a tech company</a></p></div><div><p>08/19</p><p><a href="https://www.patreon.com/posts/29534139">Buying a house as an investment</a></p></div><div><p>07/19</p><p><a href="https://www.patreon.com/posts/27400033">Kyle Petty on Danica Patrick</a></p></div><div><p>06/19</p><p><a href="https://www.patreon.com/posts/27662175">The "production ready" duality</a></p></div><div><p>06/19</p><p><a href="https://www.patreon.com/posts/27548525">Files, the talk</a></p></div><div><p>06/19</p><p><a href="https://www.patreon.com/posts/27316256">A surprising thing (to me) about bay area groupthink</a></p></div><div><p>05/19</p><p><a href="https://www.patreon.com/posts/26585592">Baseball scouting reports</a></p></div><div><p>05/19</p><p><a href="https://www.patreon.com/posts/26272280">"Caisson", the company</a></p></div><div><p>04/19</p><p><a href="https://www.patreon.com/posts/26207122">Interviews in practice, part 2</a></p></div><div><p>04/19</p><p><a href="https://www.patreon.com/posts/25835707">A puzzle about prestige</a></p></div><div><p>03/19</p><p><a href="https://www.patreon.com/posts/25016740">Dustin Curtis and Donald Knuth</a></p></div><div><p>02/19</p><p><a href="https://www.patreon.com/posts/24780296">Algorithms interviews in practice</a></p></div><div><p>12/18</p><p><a href="https://www.patreon.com/posts/23228622">What's so bad about JIRA?</a></p></div><div><p>11/18</p><p><a href="https://www.patreon.com/posts/22335083">The fallacy of corporate intent</a></p></div><div><p>10/18</p><p><a href="https://www.patreon.com/posts/22091116">Willful ignorance in management</a></p></div><div><p>09/18</p><p><a href="https://www.patreon.com/posts/21164582">What's up with Google's Spanner SLA?</a></p></div><div><p>08/18</p><p><a href="https://www.patreon.com/posts/20571244">What's going on with Qualcomm's server team?</a></p></div><div><p>08/18</p><p><a href="https://www.patreon.com/posts/19337680">Why is bridge declining?</a></p></div><div><p>05/18</p><p><a href="https://www.patreon.com/posts/18647973">Online games after a 20 year break</a></p></div><div><p>03/18</p><p><a href="https://www.patreon.com/posts/17875180">On being in it for the money</a></p></div><div><p>02/18</p><p><a href="https://www.patreon.com/posts/17154261">We can move as fast as any startup</a></p></div><div><p>01/18</p><p><a href="https://www.patreon.com/posts/16548448">Linus's rants</a></p></div><div><p>01/18</p><p><a href="https://www.patreon.com/posts/16164634">Internet comments</a></p></div><div><p>12/17</p><p><a href="https://www.patreon.com/posts/15719767">Information asymmetry plus monopsony hiring power</a></p></div><div><p>11/17</p><p><a href="https://www.patreon.com/posts/15526120">Concision vs. pre-emptive nitpicking defenses</a></p></div><div><p>10/17</p><p><a href="https://www.patreon.com/posts/15103841">An unintended consequence of company propoganda</a></p></div><div><p>08/17</p><p><a href="https://www.patreon.com/posts/13543653">What do interview questions actually measure?</a></p></div><div><p>07/17</p><p><a href="https://www.patreon.com/posts/13449675">Bootcamp vs. College</a></p></div><div><p>07/17</p><p><a href="https://www.patreon.com/posts/13169939">Attrition</a></p></div><div><p>07/17</p><p><a href="https://www.patreon.com/posts/11646938">Relative vs. absolute</a></p></div><div><p>06/17</p><p><a href="https://www.patreon.com/posts/11610960">Patreon "blog"</a></p></div></div>||||I|||| xx/xx
 Patreon posts
 09/22
 Futurist prediction methods and accuracy
 04/22
 In defense of simple architectures
 03/22
 Why is it so hard to buy things that work well?
 02/22
 Misidentifying talent
 02/22
 A decade of major cache incidents at Twitter
 02/22
 Cocktail party ideas
 12/21
 The container throttling problem
 12/21
 Some thoughts on writing
 12/21
 Some latency measurement pitfalls
 11/21
 Major errors on this blog (and their corrections)
 11/21
 Individuals matter
 11/21
 Culture matters
 10/21
 Willingness to look stupid
 10/21
 What to learn
 10/21
 Some reasons to work on productivity and velocity
 09/21
 The value of in-house expertise
 08/21
 Some reasons to measure
 12/20
 Against essential and accidental complexity
 06/20
 How do cars fare in crash tests they're not specifically optimized for?
 05/20
 A simple way to get more value from tracing
 05/20
 A simple way to get more value from metrics
 03/20
 How (some) good corporate engineering blogs are written
 03/20
 The growth of command line options, 1979-Present
 02/20
 Suspicious discontinuities
 02/20
 95%-ile isn't that good
 01/20
 Algorithms interviews: theory vs. practice
 07/19
 Files are fraught with peril
 02/19
 Randomized trial on gender in Overwatch
 12/17
 Computer latency: 1977-2017
 11/17
 How good are decisions?
 11/17
 How out of date are Android devices?
 11/17
 UI backwards compatibility
 10/17
 Filesystem error handling
 10/17
 Keyboard latency
 08/17
 Branch prediction
 08/17
 Sattolo's algorithm
 07/17
 Terminal latency
 06/17
 Keyboard v. mouse
 06/17
 Startup options v. cash
 02/17
 The modern web on a slow connection
 10/16
 HN: the good parts
 10/16
 Programming book recommendations and anti-recommendations
 10/16
 Hiring and the market for lemons
 10/16
 I could do that in a weekend!
 09/16
 Is dev compensation bimodal?
 09/16
 How I learned to program
 08/16
 Notes on concurrency bugs
 04/16
 Some programming blogs to consider reading
 04/16
 Google SRE book
 03/16
 We only hire the trendiest
 01/16
 Sampling v. tracing
 01/16
 CPU bugs
 12/15
 Normalization of deviance
 12/15
 Big companies v. startups
 12/15
 Files are hard
 11/15
 Why use ECC?
 11/15
 What's worked in Computer Science: 1999 v. 2015
 11/15
 Infinite disk
 10/15
 Why Intel added cache partitioning
 09/15
 Slowlock
 08/15
 Steve Yegge's prediction record
 08/15
 Reading postmortems
 05/15
 Slashdot and Sourceforge
 05/15
 The googlebot monopoly
 05/15
 A defense of boring languages
 05/15
 Advantages of monorepos
 03/15
 Reading citations is easier than most people think
 03/15
 Given that devs spend the effort they do on testing, what can we do to improve testing?
 03/15
 What happens when you load a URL?
 03/15
 Goodhearting IQ, cholesterol, and tail latency
 02/15
 AI doesn't have to be that good
 02/15
 CPU backdoors
 01/15
 Blog monetization
 01/15
 What's new in CPUs since the 80s?
 12/14
 A review of the Julia language
 12/14
 Integer overflow checking cost
 12/14
 Malloc tutorial
 12/14
 Markets, discrimination, and "lowering the bar"
 11/14
 TF-IDF linux commits
 11/14
 One week of bugs
 11/14
 Speeding up this site by 50x
 11/14
 Build uptime
 11/14
 Literature review on the benefits of static types
 11/14
 CLWB and PCOMMIT
 11/14
 Caches: LRU v. random
 11/14
 Testing v. informal reasoning
 10/14
 Assembly v. intrinsics
 04/14
 Bug analytics
 03/14
 Editing binaries
 03/14
 That bogus gender gap article
 03/14
 A solution to DeWitt clauses
 02/14
 Why don't schools teach debugging?
 01/14
 Do programmers need math?
 01/14
 Data alignment and caches
 12/13
 PCA is not a panacea
 11/13
 The complexity of knowledge and skill transfer
 10/13
 How to discourage open source contributions
 10/13
 Randomize HN
 09/13
 Writing safe Verilog
 09/13
 Verilog is weird
 ↑ Public posts ↓Patreon posts

 02/22
 Pop analogies
 02/22
 Ironic, self-refuting, rebuttals
 01/22
 Doing good engineering with bad mental models
 01/22
 The backlash against doing work (no, not that one)
 12/21
 My style vs. monetization
 12/21
 Phrases that mean their opposite
 11/21
 Cost of living adjustments and location-based pay
 11/21
 Writing down the most obvious things
 08/21
 On Alan Kay's claim that computers would be 1000x faster if we listened to him
 07/21
 Some reasons to measure (draft)
 03/21
 What to learn? (draft)
 01/21
 The glorification of elite teams
 09/20
 What does it mean that some Google execs think they gave away the farm with k8s?
 07/20
 You'd have to be very smart to come up with that
 06/20
 Reactions to the NYT's potential doxxing of Scott Alexander
 06/20
 Why are you still working?
 05/20
 Blog posts I'm not publishing
 02/20
 Another case against taking startup employee equity
 12/19
 Working at an ad supported company (or not)
 11/19
 Startup vs. big company compensation in practice
 09/19
 We're just like a tech company
 08/19
 Buying a house as an investment
 07/19
 Kyle Petty on Danica Patrick
 06/19
 The "production ready" duality
 06/19
 Files, the talk
 06/19
 A surprising thing (to me) about bay area groupthink
 05/19
 Baseball scouting reports
 05/19
 "Caisson", the company
 04/19
 Interviews in practice, part 2
 04/19
 A puzzle about prestige
 03/19
 Dustin Curtis and Donald Knuth
 02/19
 Algorithms interviews in practice
 12/18
 What's so bad about JIRA?
 11/18
 The fallacy of corporate intent
 10/18
 Willful ignorance in management
 09/18
 What's up with Google's Spanner SLA?
 08/18
 What's going on with Qualcomm's server team?
 08/18
 Why is bridge declining?
 05/18
 Online games after a 20 year break
 03/18
 On being in it for the money
 02/18
 We can move as fast as any startup
 01/18
 Linus's rants
 01/18
 Internet comments
 12/17
 Information asymmetry plus monopsony hiring power
 11/17
 Concision vs. pre-emptive nitpicking defenses
 10/17
 An unintended consequence of company propoganda
 08/17
 What do interview questions actually measure?
 07/17
 Bootcamp vs. College
 07/17
 Attrition
 07/17
 Relative vs. absolute
 06/17
 Patreon "blog"
RSS
xx/xx  Patreon posts    09/22  Futurist prediction methods and accuracy    04/22  In defense of simple architectures    03/22  Why is it so hard to buy things that work well?    02/22  Misidentifying talent    02/22  A decade of major cache incidents at Twitter    02/22  Cocktail party ideas    12/21  The container throttling problem    12/21  Some thoughts on writing    12/21  Some latency measurement pitfalls    11/21  Major errors on this blog (and their corrections)    11/21  Individuals matter    11/21  Culture matters    10/21  Willingness to look stupid    10/21  What to learn    10/21  Some reasons to work on productivity and velocity    09/21  The value of in-house expertise    08/21  Some reasons to measure    12/20  Against essential and accidental complexity    06/20  How do cars fare in crash tests they're not specifically optimized for?    05/20  A simple way to get more value from tracing    05/20  A simple way to get more value from metrics    03/20  How (some) good corporate engineering blogs are written    03/20  The growth of command line options, 1979-Present    02/20  Suspicious discontinuities    02/20  95%-ile isn't that good    01/20  Algorithms interviews: theory vs. practice    07/19  Files are fraught with peril    02/19  Randomized trial on gender in Overwatch    12/17  Computer latency: 1977-2017    11/17  How good are decisions?    11/17  How out of date are Android devices?    11/17  UI backwards compatibility    10/17  Filesystem error handling    10/17  Keyboard latency    08/17  Branch prediction    08/17  Sattolo's algorithm    07/17  Terminal latency    06/17  Keyboard v. mouse    06/17  Startup options v. cash    02/17  The modern web on a slow connection    10/16  HN: the good parts    10/16  Programming book recommendations and anti-recommendations    10/16  Hiring and the market for lemons    10/16  I could do that in a weekend!    09/16  Is dev compensation bimodal?    09/16  How I learned to program    08/16  Notes on concurrency bugs    04/16  Some programming blogs to consider reading    04/16  Google SRE book    03/16  We only hire the trendiest    01/16  Sampling v. tracing    01/16  CPU bugs    12/15  Normalization of deviance    12/15  Big companies v. startups    12/15  Files are hard    11/15  Why use ECC?    11/15  What's worked in Computer Science: 1999 v. 2015    11/15  Infinite disk    10/15  Why Intel added cache partitioning    09/15  Slowlock    08/15  Steve Yegge's prediction record    08/15  Reading postmortems    05/15  Slashdot and Sourceforge    05/15  The googlebot monopoly    05/15  A defense of boring languages    05/15  Advantages of monorepos    03/15  Reading citations is easier than most people think    03/15  Given that devs spend the effort they do on testing, what can we do to improve testing?    03/15  What happens when you load a URL?    03/15  Goodhearting IQ, cholesterol, and tail latency    02/15  AI doesn't have to be that good    02/15  CPU backdoors    01/15  Blog monetization    01/15  What's new in CPUs since the 80s?    12/14  A review of the Julia language    12/14  Integer overflow checking cost    12/14  Malloc tutorial    12/14  Markets, discrimination, and "lowering the bar"    11/14  TF-IDF linux commits    11/14  One week of bugs    11/14  Speeding up this site by 50x    11/14  Build uptime    11/14  Literature review on the benefits of static types    11/14  CLWB and PCOMMIT    11/14  Caches: LRU v. random    11/14  Testing v. informal reasoning    10/14  Assembly v. intrinsics    04/14  Bug analytics    03/14  Editing binaries    03/14  That bogus gender gap article    03/14  A solution to DeWitt clauses    02/14  Why don't schools teach debugging?    01/14  Do programmers need math?    01/14  Data alignment and caches    12/13  PCA is not a panacea    11/13  The complexity of knowledge and skill transfer    10/13  How to discourage open source contributions    10/13  Randomize HN    09/13  Writing safe Verilog    09/13  Verilog is weird    ↑ Public posts    ↓Patreon posts  02/22  Pop analogies    02/22  Ironic, self-refuting, rebuttals    01/22  Doing good engineering with bad mental models    01/22  The backlash against doing work (no, not that one)    12/21  My style vs. monetization    12/21  Phrases that mean their opposite    11/21  Cost of living adjustments and location-based pay    11/21  Writing down the most obvious things    08/21  On Alan Kay's claim that computers would be 1000x faster if we listened to him    07/21  Some reasons to measure (draft)    03/21  What to learn? (draft)    01/21  The glorification of elite teams    09/20  What does it mean that some Google execs think they gave away the farm with k8s?    07/20  You'd have to be very smart to come up with that    06/20  Reactions to the NYT's potential doxxing of Scott Alexander    06/20  Why are you still working?    05/20  Blog posts I'm not publishing    02/20  Another case against taking startup employee equity    12/19  Working at an ad supported company (or not)    11/19  Startup vs. big company compensation in practice    09/19  We're just like a tech company    08/19  Buying a house as an investment    07/19  Kyle Petty on Danica Patrick    06/19  The "production ready" duality    06/19  Files, the talk    06/19  A surprising thing (to me) about bay area groupthink    05/19  Baseball scouting reports    05/19  "Caisson", the company    04/19  Interviews in practice, part 2    04/19  A puzzle about prestige    03/19  Dustin Curtis and Donald Knuth    02/19  Algorithms interviews in practice    12/18  What's so bad about JIRA?    11/18  The fallacy of corporate intent    10/18  Willful ignorance in management    09/18  What's up with Google's Spanner SLA?    08/18  What's going on with Qualcomm's server team?    08/18  Why is bridge declining?    05/18  Online games after a 20 year break    03/18  On being in it for the money    02/18  We can move as fast as any startup    01/18  Linus's rants    01/18  Internet comments    12/17  Information asymmetry plus monopsony hiring power    11/17  Concision vs. pre-emptive nitpicking defenses    10/17  An unintended consequence of company propoganda    08/17  What do interview questions actually measure?    07/17  Bootcamp vs. College    07/17  Attrition    07/17  Relative vs. absolute    06/17  Patreon "blog"
https://danluu.com/
https://medium.com/@dschmidt1992
Daniel Schmidt – Medium
2017-12-09
Daniel Schmidt
<div><div><a href="https://medium.com/react-native-training/building-a-react-native-game-with-rxjs-5-ced0e716bc73?source=user_profile---------6----------------------------"><h2>Building a React Native Game with RxJS 5</h2><p><h3>Lately at work, my colleagues build some awesome tooling to fetch data in an easy way. It’s built around RxJS, which I previously had very…</h3></p></a></div></div>||||I|||| Open in app

 Sign up

 Sign in

 Write

 Sign up

 Sign in

 Daniel Schmidt

 674 Followers

 Home

 About

 Daniel Schmidt

 Implementing GraphQL Subscriptions with RxJS

 I recently had to bootstrap a project with a GraphQL frontend and backend using Apollo. We wanted to show realtime updates of metrics we…

 1 min read · Jul 19, 2019

 --

 --

 Daniel Schmidt

 in

 React Native Training

 Making your React Native development environment scalable for Teams

 “It works on my machine” is something you don’t want to experience in your team. This is especially true when working on a React Native…

 2 min read · Oct 12, 2018

 --

 --

 Daniel Schmidt

 in

 HackerNoon.com

 Performance Tracing for GraphQL with Apollo and it’s friends

 Often when you develop an application performance matters. The most important asset you have when it comes to perceived performance is the…

 3 min read · Sep 13, 2018

 --

 1

 --

 1

 Daniel Schmidt

 in

 React Native Training

 Build YouTube alike Livestreams with React Native

 After building an intro on how to upload videos with React Native I would like to go a step further and build live streaming. For me this…

 5 min read · Jul 24, 2018

 --

 8

 --

 8

 Daniel Schmidt

 in

 React Native Training

 Ignite 2 — The shortcut for new React Native projects

 Almost two years ago I gave Ignite by Infinite Red a try and I was instantly hooked. It was a super convenient and fast way to start a new…

 4 min read · Jul 16, 2018

 --

 3

 --

 3

 Daniel Schmidt

 in

 React Native Training

 Record and Upload Videos with React Native

 In one of my last blog posts, Samuel Omole asked me if I could do an article about uploading content with React Native. For me the most…

 4 min read · Jul 10, 2018

 --

 24

 --

 24

 Daniel Schmidt

 in

 React Native Training

 Building a React Native Game with RxJS 5

 Lately at work, my colleagues build some awesome tooling to fetch data in an easy way. It’s built around RxJS, which I previously had very…

 4 min read · Jun 12, 2018

 --

 3

 --

 3

 Daniel Schmidt

 in

 React Native Training

 Using Sensors in React Native

 One of the most exciting strengths of React Native is that it allows you to bridge native calls to Javascript. In contrast to other…

 4 min read · May 14, 2018

 --

 9

 --

 9

 Daniel Schmidt

 My productivity setup

 My co-worker Fabs recently wrote an article about how he get’s work done, which inspired me to write down my recent learnings about my own…

 6 min read · Mar 6, 2018

 --

 3

 --

 3

 Daniel Schmidt

 in

 HackerNoon.com

 Performance Monitoring for the Frontend — Using Zipkin

 This series of blog posts is about the network related performance issues. The goal is to give you a set of tools to use that will provide…

 3 min read · Dec 9, 2017

 --

 1

 --

 1

 Daniel Schmidt

 674 Followers

 Software Engineer at Mesosphere working on the DC/OS UI

 Follow
 Following

 * Ivan Sim

 * Open GraphQL

 * Marv

 * Weston Bassler

 * React Native Training

 See all (199)

 Help

 Status

 About

 Careers

 Blog

 Privacy

 Terms

 Text to speech

 Teams

Building a React Native Game with RxJS 5  Lately at work, my colleagues build some awesome tooling to fetch data in an easy way. It’s built around RxJS, which I previously had very…
Daniel Schmidt – Medium
https://medium.com/@dschmidt1992
https://datchley.name/
Datchley's News
2023-02-06
None
<div><div><div><h2><a href="https://datchley.name/a-little-catch-up/">A little Catch-Up</a></h2><p>26 January, 2023</p><p></p><p> Where on earth have you been Clodagh?! So there's this guy…. that i've been spending a lot of time with lately and yeah, I guess I haven't had much time for anything else. I'm sure you don't want to hear any v...</p></div><div><h2><a href="https://datchley.name/best-private-wow-servers/">Best private wow servers</a></h2><p>19 January, 2023</p><p></p><p> Choosing the right server for you can be quite tricky, however here are my favorite servers, to help you out if you're not sure about looking for them yourself. Obviously the best way to go about it yourself i...</p></div><div><h2><a href="https://datchley.name/interview-claudian/">Interview: Claudian</a></h2><p>14 January, 2023</p><p></p><p> Folks, today I am joined by none other than Claudian, the Game Master Manager here at Feenix. Today we delve into the deep murky waters of Claudian's mind and discover what lurks beneath… Brace yourselves, the...</p></div><div><h2><a href="https://datchley.name/interview-sideways/">Interview: Sideways</a></h2><p>5 January, 2023</p><p></p><p> Ever wondered what the head developer on Feenix gets up to on a daily basis? How much work is involved or how he got started on the project? Perhaps some insight into the new Emerald Dream realm which is due f...</p></div><div><h2><a href="https://datchley.name/new-vanilla-wow-realm/">New Vanilla WoW Realm</a></h2><p>1 January, 2023</p><p></p><p> So unless you've been hiding under a rock for a few months, you would know that Feenix is planning a delicious new Vanilla World of Warcraft Realm. I've put together a brief FAQ together with my own hopes for ...</p></div><div><h2><a href="https://datchley.name/vanilla-wow-addons/">Vanilla WoW Addons</a></h2><p>25 December, 2022</p><p></p><p> So you've just rolled a brand new char. How much time do you spend sorting out the addons? Are you one of the people who prefer to use the basic blizzard ui or are you one of the people who spend days making y...</p></div><div><h2><a href="https://datchley.name/wotlk-or-wrath-choice/">WotLK or Wrath choice?</a></h2><p>11 December, 2022</p><p></p><p> Hey everyone, I'm looking for the best choice for me in terms of WotLK. Ideally I'd go for a server still in its progression infancy but alas, no such luck, everyone's at ICC. So I slimmed down my choices do D...</p></div><div><h2><a href="https://datchley.name/wow-private-server-faq/">WoW private server faq</a></h2><p>28 November, 2022</p><p></p><p> What are WoW private servers ?Wow servers, also known as realms, allow you to play world of warcraft. Private servers allow you to play on a server curated by someone other than blizzard. Is it legal ?Short an...</p></div><div><h2><a href="https://datchley.name/wow-private-servers/">Wow private servers</a></h2><p>25 November, 2022</p><p></p><p> I was extremely hyped for the release of the vanilla wow server. It brought out my nostalgia and after playing it for a while now I've been thoroughly enjoying it. However lately it has gotten a little stale a...</p></div><div><h2><a href="https://datchley.name/wow-server-throttling/">Wow Server Throttling</a></h2><p>13 November, 2022</p><p></p><p> A key difference between private servers long ago and private servers today was the shear quantity of them. When I first started playing on private servers (Wrath was still early into its expansion cycle) ther...</p></div></div>||||I|||| Datchley's News

 A little Catch-Up

 26 January, 2023

 Where on earth have you been Clodagh?! So there's this guy…. that i've been spending a lot of time with lately and yeah, I guess I haven't had much time for anything else. I'm sure you don't want to hear any v...

 Best private wow servers

 19 January, 2023

 Choosing the right server for you can be quite tricky, however here are my favorite servers, to help you out if you're not sure about looking for them yourself. Obviously the best way to go about it yourself i...

 Interview: Claudian

 14 January, 2023

 Folks, today I am joined by none other than Claudian, the Game Master Manager here at Feenix. Today we delve into the deep murky waters of Claudian's mind and discover what lurks beneath… Brace yourselves, the...

 Interview: Sideways

 5 January, 2023

 Ever wondered what the head developer on Feenix gets up to on a daily basis? How much work is involved or how he got started on the project? Perhaps some insight into the new Emerald Dream realm which is due f...

 New Vanilla WoW Realm

 1 January, 2023

 So unless you've been hiding under a rock for a few months, you would know that Feenix is planning a delicious new Vanilla World of Warcraft Realm. I've put together a brief FAQ together with my own hopes for ...

 Vanilla WoW Addons

 25 December, 2022

 So you've just rolled a brand new char. How much time do you spend sorting out the addons? Are you one of the people who prefer to use the basic blizzard ui or are you one of the people who spend days making y...

 WotLK or Wrath choice?

 11 December, 2022

 Hey everyone, I'm looking for the best choice for me in terms of WotLK. Ideally I'd go for a server still in its progression infancy but alas, no such luck, everyone's at ICC. So I slimmed down my choices do D...

 WoW private server faq

 28 November, 2022

 What are WoW private servers ?Wow servers, also known as realms, allow you to play world of warcraft. Private servers allow you to play on a server curated by someone other than blizzard. Is it legal ?Short an...

 Wow private servers

 25 November, 2022

 I was extremely hyped for the release of the vanilla wow server. It brought out my nostalgia and after playing it for a while now I've been thoroughly enjoying it. However lately it has gotten a little stale a...

 Wow Server Throttling

 13 November, 2022

 A key difference between private servers long ago and private servers today was the shear quantity of them. When I first started playing on private servers (Wrath was still early into its expansion cycle) ther...

 datchley.name - 6 February, 2023

A little Catch-Up  26 January, 2023    Where on earth have you been Clodagh?! So there's this guy…. that i've been spending a lot of time with lately and yeah, I guess I haven't had much time for anything else. I'm sure you don't want to hear any v...    Best private wow servers  19 January, 2023    Choosing the right server for you can be quite tricky, however here are my favorite servers, to help you out if you're not sure about looking for them yourself. Obviously the best way to go about it yourself i...    Interview: Claudian  14 January, 2023    Folks, today I am joined by none other than Claudian, the Game Master Manager here at Feenix. Today we delve into the deep murky waters of Claudian's mind and discover what lurks beneath… Brace yourselves, the...    Interview: Sideways  5 January, 2023    Ever wondered what the head developer on Feenix gets up to on a daily basis? How much work is involved or how he got started on the project? Perhaps some insight into the new Emerald Dream realm which is due f...    New Vanilla WoW Realm  1 January, 2023    So unless you've been hiding under a rock for a few months, you would know that Feenix is planning a delicious new Vanilla World of Warcraft Realm. I've put together a brief FAQ together with my own hopes for ...    Vanilla WoW Addons  25 December, 2022    So you've just rolled a brand new char. How much time do you spend sorting out the addons? Are you one of the people who prefer to use the basic blizzard ui or are you one of the people who spend days making y...    WotLK or Wrath choice?  11 December, 2022    Hey everyone, I'm looking for the best choice for me in terms of WotLK. Ideally I'd go for a server still in its progression infancy but alas, no such luck, everyone's at ICC. So I slimmed down my choices do D...    WoW private server faq  28 November, 2022    What are WoW private servers ?Wow servers, also known as realms, allow you to play world of warcraft. Private servers allow you to play on a server curated by someone other than blizzard. Is it legal ?Short an...    Wow private servers  25 November, 2022    I was extremely hyped for the release of the vanilla wow server. It brought out my nostalgia and after playing it for a while now I've been thoroughly enjoying it. However lately it has gotten a little stale a...    Wow Server Throttling  13 November, 2022    A key difference between private servers long ago and private servers today was the shear quantity of them. When I first started playing on private servers (Wrath was still early into its expansion cycle) ther...
Datchley's News
http://www.datchley.name/
https://dabeaz.com/blog.html
Dave's Blog
2009-08-09
None
<div><div>
 <h2>Dave's Blog</h2>
 
 <p>
 <b>Current Entry</b>
 </p>
 <ul>
 <li>07 Jan 2015. <a href="blog/5k/index.html">Running a 5K Versus Field Day</a></li>
 </ul>
 
 <h3>Archive of Posts (by topic)</h3>
 
 <p>
 Python3 and Python Cookbook
 </p>
 <ul>
 <li>19 Jan 2011. <a href="http://dabeaz.blogspot.com/2011/01/porting-py65-and-my-superboard-to.html">Porting Py65 (and my Superboard) to Python 3</a></li>
 <li>16 Dec 2010. <a href="http://dabeaz.blogspot.com/2010/12/oreilly-python-cookbook-python-3-all.html">
 O'Reilly Python Cookbook: Python 3 All The Way</a></li>
 </ul>
 
 <p>
 Python Concurrency:
 </p>
 <ul>
 <li>31 Jul 2010. <a href="http://dabeaz.blogspot.com/2010/07/yieldable-threads-part-1.html">Yieldable Threads (Part 1)</a></li>
 <li>20 Nov 2009. <a href="http://dabeaz.blogspot.com/2009/11/python-thread-deadlock-avoidance_20.html">Python Thread Deadlock Avoidance</a></li>
 <li>13 Sep 2009. <a href="http://dabeaz.blogspot.com/2009/09/python-thread-synchronization.html">Python Thread Synchronization Primitives : Not Entirely What You Think</a></li>
 </ul>
 
 <p>
 The GIL:
 </p>
 
 <ul>
 <li>12 Aug 2011. <a href="http://dabeaz.blogspot.com/2011/08/inside-look-at-gil-removal-patch-of.html">An Inside Look at the GIL Removal Patch of Lore</a></li>
 <li>22 Feb 2010. <a href="http://dabeaz.blogspot.com/2010/02/revisiting-thread-priorities-and-new.html">Revisiting thread priorities and the new GIL</a></li>
 <li>17 Jan 2010. <a href="http://dabeaz.blogspot.com/2010/01/presentation-on-new-python-gil.html">Presentation on the new Python GIL</a></li>
 <li>05 Jan 2010. <a href="http://dabeaz.blogspot.com/2010/01/python-gil-visualized.html">The Python GIL Visualized</a></li>
 <li>27 Aug 2009. <a href="http://dabeaz.blogspot.com/2009/08/inside-inside-python-gil-presentation.html">Inside the "Inside the Python GIL" Presentation</a></li>
 </ul>
 
 <p>
 Python I/O:
 </p>
 
 <ul>
 <li>28 Jan 2010. <a href="http://dabeaz.blogspot.com/2010/01/few-useful-bytearray-tricks.html">A few useful bytearray tricks</a></li>
 <li>26 Jan 2010. <a href="http://dabeaz.blogspot.com/2010/01/reexamining-python-3-text-io.html">Reexamining Python 3 Text I/O</a></li>
 <li>09 Aug 2009. <a href="http://dabeaz.blogspot.com/2009/08/python-binary-io-handling.html">Python Binary I/O Handling</a></li>
 </ul>
 
 <p>
 Python Programming (General):
 </p>
 <ul>
 <li>02 Feb 2010. <a href="http://dabeaz.blogspot.com/2010/02/function-that-works-as-context-manager.html">A function that works as a context manager and a decorator</a></li>
 <li>02 Feb 2010. <a href="http://dabeaz.blogspot.com/2010/02/context-manager-for-timing-benchmarks.html">A Context Manager for Timing Benchmarks</a></li>
 </ul>
 
 <p>
 Python Essential Reference:
 </p>
 
 <ul>
 <li>21 Jan 2010. <a href="http://dabeaz.blogspot.com/2010/01/slashdot-pronouns-and-python-essential.html">Slashdot, Pronouns, and the Python Essential Reference</a></li>
 <li>09 Aug 2009. <a href="http://dabeaz.blogspot.com/2009/08/essential-misconceptions.html">Essential Misconceptions</a></li>
 </ul>
 
 <p>
 Superboard II:
 </p>
 <ul>
 <li>15 Mar 2011. <a href="http://dabeaz.blogspot.com/2011/03/superboard-takes-pycon.html">The Superboard Takes Pycon!</a></li>
 <li>19 Jan 2011. <a href="http://dabeaz.blogspot.com/2011/01/porting-py65-and-my-superboard-to.html">Porting Py65 (and my Superboard) to Python 3</a></li>
 <li>04 Sep 2010. <a href="http://dabeaz.blogspot.com/2010/09/using-telnet-to-access-my-superboard-ii.html">Using telnet to access my Superboard II (via Python and cassette ports)</a></li>
 <li>29 Aug 2010. <a href="http://dabeaz.blogspot.com/2010/08/decoding-superboard-ii-cassette-audio.html">Decoding Superboard II Cassette Audio Using Python 3, Two Generators, and a Deque</a></li>
 <li>22 Aug 2010. <a href="http://dabeaz.blogspot.com/2010/08/using-python-to-encode-cassette.html">Using Python to Encode Cassette Recordings for my Superboard II</a></li>
 </ul>
 
 <p>
 For Fun:
 </p>
 
 <ul>
 <li>07 Jan 2015. <a href="blog/5k/index.html">Running a 5K Versus Field Day</a></li>
 <li>02 Jan 2015. <a href="blog/Bike2014/bike.html">On Early Morning Biking Through a Chicago Year</a></li>
 <li>31 Oct 2009. <a href="http://dabeaz.blogspot.com/2009/10/ultimate-python-quickstart-guide.html">Ultimate Python Quickstart Guide</a></li>
 <li>27 Nov 2009. <a href="http://dabeaz.blogspot.com/2009/11/fun-with-block-towers.html">Fun With Block Towers</a></li>
 
 </ul>
</div>||||I||||dabeaz
 * Courses
 * Writing
 Python Distilled Python Essential Reference Python Cookbook Usenix :login; Academic Publications
 * Speaking
 Talks Tutorials
 * Software
 * About
 * Newsletter
 * Chat
 Contact

 Dave's Blog

 Current Entry

 * 07 Jan 2015. Running a 5K Versus Field Day

 Archive of Posts (by topic)

 Python3 and Python Cookbook

 * 19 Jan 2011. Porting Py65 (and my Superboard) to Python 3
 * 16 Dec 2010. O'Reilly Python Cookbook: Python 3 All The Way

 Python Concurrency:

 * 31 Jul 2010. Yieldable Threads (Part 1)
 * 20 Nov 2009. Python Thread Deadlock Avoidance
 * 13 Sep 2009. Python Thread Synchronization Primitives : Not Entirely What You Think

 The GIL:

 * 12 Aug 2011. An Inside Look at the GIL Removal Patch of Lore
 * 22 Feb 2010. Revisiting thread priorities and the new GIL
 * 17 Jan 2010. Presentation on the new Python GIL
 * 05 Jan 2010. The Python GIL Visualized
 * 27 Aug 2009. Inside the "Inside the Python GIL" Presentation

 Python I/O:

 * 28 Jan 2010. A few useful bytearray tricks
 * 26 Jan 2010. Reexamining Python 3 Text I/O
 * 09 Aug 2009. Python Binary I/O Handling

 Python Programming (General):

 * 02 Feb 2010. A function that works as a context manager and a decorator
 * 02 Feb 2010. A Context Manager for Timing Benchmarks

 Python Essential Reference:

 * 21 Jan 2010. Slashdot, Pronouns, and the Python Essential Reference
 * 09 Aug 2009. Essential Misconceptions

 Superboard II:

 * 15 Mar 2011. The Superboard Takes Pycon!
 * 19 Jan 2011. Porting Py65 (and my Superboard) to Python 3
 * 04 Sep 2010. Using telnet to access my Superboard II (via Python and cassette ports)
 * 29 Aug 2010. Decoding Superboard II Cassette Audio Using Python 3, Two Generators, and a Deque
 * 22 Aug 2010. Using Python to Encode Cassette Recordings for my Superboard II

 For Fun:

 * 07 Jan 2015. Running a 5K Versus Field Day
 * 02 Jan 2015. On Early Morning Biking Through a Chicago Year
 * 31 Oct 2009. Ultimate Python Quickstart Guide
 * 27 Nov 2009. Fun With Block Towers
Dave's Blog
 
 
 Current Entry
 
 
 07 Jan 2015. Running a 5K Versus Field Day 
 
 
 Archive of Posts (by topic)
 
 
 Python3 and Python Cookbook
 
 
 19 Jan 2011. Porting Py65 (and my Superboard) to Python 3 
 16 Dec 2010. 
 O'Reilly Python Cookbook: Python 3 All The Way 
 
 
 
 Python Concurrency:
 
 
 31 Jul 2010. Yieldable Threads (Part 1) 
 20 Nov 2009. Python Thread Deadlock Avoidance 
 13 Sep 2009. Python Thread Synchronization Primitives : Not Entirely What You Think 
 
 
 
 The GIL:
 
 
 
 12 Aug 2011. An Inside Look at the GIL Removal Patch of Lore 
 22 Feb 2010. Revisiting thread priorities and the new GIL 
 17 Jan 2010. Presentation on the new Python GIL 
 05 Jan 2010. The Python GIL Visualized 
 27 Aug 2009. Inside the "Inside the Python GIL" Presentation 
 
 
 
 Python I/O:
 
 
 
 28 Jan 2010. A few useful bytearray tricks 
 26 Jan 2010. Reexamining Python 3 Text I/O 
 09 Aug 2009. Python Binary I/O Handling 
 
 
 
 Python Programming (General):
 
 
 02 Feb 2010. A function that works as a context manager and a decorator 
 02 Feb 2010. A Context Manager for Timing Benchmarks 
 
 
 
 Python Essential Reference:
 
 
 
 21 Jan 2010. Slashdot, Pronouns, and the Python Essential Reference 
 09 Aug 2009. Essential Misconceptions 
 
 
 
 Superboard II:
 
 
 15 Mar 2011. The Superboard Takes Pycon! 
 19 Jan 2011. Porting Py65 (and my Superboard) to Python 3 
 04 Sep 2010. Using telnet to access my Superboard II (via Python and cassette ports) 
 29 Aug 2010. Decoding Superboard II Cassette Audio Using Python 3, Two Generators, and a Deque 
 22 Aug 2010. Using Python to Encode Cassette Recordings for my Superboard II 
 
 
 
 For Fun:
 
 
 
 07 Jan 2015. Running a 5K Versus Field Day 
 02 Jan 2015. On Early Morning Biking Through a Chicago Year 
 31 Oct 2009. Ultimate Python Quickstart Guide 
 27 Nov 2009. Fun With Block Towers
Dave's Blog
http://www.dabeaz.com/blog.html
https://dave.cheney.net/
A few bytes here, a few there, pretty soon you’re talking real memory
2021-01-05
Dave Cheney
<div><div>
	<article>
	<div>
	
<p>Today’s post comes from a recent Go pop quiz. Consider this benchmark fragment.<a href="https://dave.cheney.net/2021/01/05/a-few-bytes-here-a-few-there-pretty-soon-youre-talking-real-memory#easy-footnote-bottom-1-4231"><sup>1</sup></a></p>
<pre><code>func BenchmarkSortStrings(b *testing.B) {
 s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"}
 b.ReportAllocs()
 for i := 0; i &lt; b.N; i++ {
 sort.Strings(s)
 }
}</code></pre>
<p>A convenience wrapper around <code>sort.Sort(sort.StringSlice(s))</code>, <code>sort.Strings</code> sorts the input in place, so it isn’t expected to allocate (or at least that’s what 43% of the tweeps who responded thought). However it turns out that, at least in recent versions of Go, each iteration of the benchmark causes one heap allocation. Why does this happen?</p>
<p>Interfaces, as all Go programmers should know, are implemented as a<a href="https://research.swtch.com/interfaces"> two word structure</a>. Each interface value contains a field which holds the type of the interface’s contents, and a pointer to the interface’s contents.<a href="https://dave.cheney.net/2021/01/05/a-few-bytes-here-a-few-there-pretty-soon-youre-talking-real-memory#easy-footnote-bottom-2-4231"><sup>2</sup></a></p>
<p>In pseudo Go code, an interface might look something like this:</p>
<pre><code>type interface struct {
 // the ordinal number for the type of the value
 // assigned to the interface 
 type uintptr
 // (usually) a pointer to the value assigned to
 // the interface
 data uintptr
}</code></pre>
<p><code>interface.data</code> can hold one machine word–8 bytes in most cases–but a <code>[]string</code> is 24 bytes; one word for the pointer to the slice’s underlying array; one word for the length; and one for the remaining capacity of the underlying array, so how does Go fit 24 bytes into 8? With the oldest trick in the book, indirection. <code>s</code>, a <code>[]string</code> is 24 bytes, but <code>*[]string</code>–a pointer to a slice of strings–is only 8. </p>
<h2>Escaping to the heap</h2>
<p>To make the example a little more explicit, here’s the benchmark rewritten without the <code>sort.Strings</code> helper function:</p>
<pre><code>func BenchmarkSortStrings(b *testing.B) {
 s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"}
 b.ReportAllocs()
 for i := 0; i &lt; b.N; i++ {
 var ss sort.StringSlice = s
 var si sort.Interface = ss // allocation
 sort.Sort(si)
 }
}
</code></pre>
<p>To make the interface magic work, the compiler rewrites the assignment as <code>var si sort.Interface = &amp;ss</code>–the <em>address</em> of <code>ss</code> is assigned to the interface value.<a href="https://dave.cheney.net/2021/01/05/a-few-bytes-here-a-few-there-pretty-soon-youre-talking-real-memory#easy-footnote-bottom-3-4231"><sup>3</sup></a> We now have a situation where the interface value holds a pointer to <code>ss</code>, but where does it point? Where in memory does <code>ss</code> live?</p>
<p>It appears that <code>ss</code> is moved to the heap, causing the allocation that the benchmark reports.</p>
<pre> Total: 296.01MB 296.01MB (flat, cum) 99.66%
 8 . . func BenchmarkSortStrings(b *testing.B) { 
 9 . . 	s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"} 
 10 . . 	b.ReportAllocs() 
 11 . . 	for i := 0; i &lt; b.N; i++ { 
 12 . . 	var ss sort.StringSlice = s 
 13 296.01MB 296.01MB 	var si sort.Interface = ss // allocation 
 14 . . 	sort.Sort(si) 
 15 . . 	} 
 16 . . } </pre>
<p>The allocation occurs because the compiler currently cannot convince itself that <code>ss</code> outlives <code>si</code>. The general attitude amongst Go compiler hackers seems to be that <a href="https://github.com/golang/go/issues/23676">this could be improved</a>, but that’s a discussion for another time. As it stands, <code>ss</code> is allocated on the heap. Thus the question becomes, how many bytes are allocated per iteration? Why don’t we ask the <code>testing</code> package.</p>
<pre>% go test -bench=. sort_test.go
goos: darwin
goarch: amd64
cpu: Intel(R) Core(TM) i7-5650U CPU @ 2.20GHz
BenchmarkSortStrings-4 12591951 91.36 ns/op 24 B/op 1 allocs/op
PASS
ok command-line-arguments 1.260s</pre>
<p>Using Go 1.16beta1, on amd64, 24 bytes are allocated per operation.<a href="https://dave.cheney.net/2021/01/05/a-few-bytes-here-a-few-there-pretty-soon-youre-talking-real-memory#easy-footnote-bottom-4-4231"><sup>4</sup></a> However, the previous Go version, on the same platform, consumes 32 bytes per operation</p>
<pre>% go1.15 test -bench=. sort_test.go
goos: darwin
goarch: amd64
BenchmarkSortStrings-4 11453016 96.4 ns/op 32 B/op 1 allocs/op
PASS
ok command-line-arguments 1.225s</pre>
<p>This brings us, circuitously, to the subject of this post, a nifty quality of life improvement coming in Go 1.16. But before I can talk about it, I need to discuss size classes.</p>
<h2>Size classes</h2>
<p>To explain what a size class is, consider how a theoretical Go runtime could allocate 24 bytes on its heap. A simple way to do this would be keep track of all the memory allocated so far using a pointer to the last allocated byte on the heap. To allocate 24 bytes, the heap pointer is incremented by 24, and the previous value returned to the caller. As long as the code that asked for 24 bytes never writes beyond that mark this mechanism has no overhead. Sadly, in real life, memory allocators don’t just allocate memory, sometimes they have to free it.</p>
<p>Eventually the Go runtime will have to free those 24 bytes, but from the runtime’s point of view, the only thing it knows is the starting address it gave to the caller. It does not know how many bytes after that address were allocated. To permit deallocation, our hypothetical Go runtime allocator would have to record, for each allocation on the heap, its length. Where are the allocation for those lengths allocated? On the heap of course.</p>
<p>In our scenario, when the runtime wants to allocate memory, it could request a little more than it was asked for and use it to store the amount requested. For our slice example, when we asked for 24 bytes, we’d actually consume 24 bytes plus some overhead to store the number 24. How large is this overhead? It turns out the minimum amount that is practical is one word.<a href="https://dave.cheney.net/2021/01/05/a-few-bytes-here-a-few-there-pretty-soon-youre-talking-real-memory#easy-footnote-bottom-5-4231"><sup>5</sup></a></p>
<p>To record a 24 byte allocation the overhead would 8 bytes. 25% is not great, but not terrible, and as the size of the allocation goes up, the overhead would become insignificant. However, what would happen if we wanted to store just one <code>byte</code> on the heap? The overhead would be eight times the amount of data we asked for! Is there are more efficient way to allocate small amounts on the heap?</p>
<p>Rather than storing the length alongside each allocation, what if all the thing of the same size were stored together? If all the 24 byte things are stored together, then the runtime would automatically know how large they are. All the runtime needs is a single bit to indicate if a 24 byte area is in use or not. In Go these areas are known as size classes, because all the things of the same size are stored together (think school class–all the students are the same grade–not a C++ class). When the runtime needs to allocate a small amount, it does so using the smallest size class that can accomodate the allocation.</p>
<h2>Unlimited size classes for all</h2>
<p>Now we know how size classes work, the obvious question is, where are they stored? Not surprisingly, the memory for a size class comes from the heap. To minimise overhead, the runtime allocates a larger amount from the heap (usually a multiple of the system page size) then dedicates that space for allocations of a single size. But, there’s a problem.</p>
<p>The pattern of allocating a large area to store things of the same size works well<a href="https://dave.cheney.net/2021/01/05/a-few-bytes-here-a-few-there-pretty-soon-youre-talking-real-memory#easy-footnote-bottom-6-4231"><sup>6</sup></a> if there’s a fixed, preferably small, number of allocation sizes, but in a general purpose language programs can ask the runtime for an allocation of any size.<a href="https://dave.cheney.net/2021/01/05/a-few-bytes-here-a-few-there-pretty-soon-youre-talking-real-memory#easy-footnote-bottom-7-4231"><sup>7</sup></a></p>
<p>For example, imagine asking the runtime for 9 bytes. 9 bytes is an uncommon size, so it’s likely that a new size class for things of size 9 would be required. As 9 byte things are uncommon, it’s likely the remainder of the allocation, 4kb or more, would be wasted. Because of this the set of possible size classes is fixed. If a size class of the exact amount is not available, the allocation is rounded up to the next size class. In our example 9 bytes might be allocated in the 12 byte size class. The overhead of 3 unused bytes is better than an entire size class allocation which goes mostly unused.</p>
<h2>All together now</h2>
<p>This is the final piece of the puzzle. Go 1.15 did not have a 24 byte size class, so the heap allocation of <code>ss</code> was allocated in the 32 byte size class. Thanks to the work of Martin Möhrmann Go 1.16 has a 24 byte size class, which is a perfect fit for slice values assigned to interfaces.</p>
	</div>
	</article>
	<article>
	<div>
	
<p>Picture yourself, an engineer working at the hottest distributed microservices de jour, assigned to fix a bug. You jump into an unfamiliar codebase and quickly locate the line where the problem occurred. The fix is simple, just return early or substitute a default value in the case that one cannot be determined from your input. Boom, problem solved. Code compiles, fix goes out to production, and the card’s in the done pile in time for Tuesday drinks at the pub.</p>
<p>Now consider this alternative scenario. You find the problematic line and, before you make the fix, you decide to add a test so you will <em>know</em> that your fix worked, and hopefully someone will not accidentally revert it in the future.</p>
<p>You’ve figured out that some weird edge case causes that line to be called without being able to determine the right value for <code>y</code>. You can see it clearly, <code>y</code> ends up being zero so the program crashes. To write a test, you need to get the conditions that caused <code>y</code> to be zero to occur on command. There are only two parameters passed into this function, this shouldn’t be hard. Oh, but this is a method, not a function, so you need to construct an instance of the object in the right state to trigger the bug. Hmm, this language uses constructors to make sure people can’t just monkey up the bits of the object they need for a test, you’ll have to find, mock, stub or build instances of all the objects dependencies (and their dependencies, and so on) then run the object through the precise series of operations to put the it in the state that causes <code>y</code> to be empty. Then you can write the test.</p>
<p>At this point Tuesday night drinks are fading before you eyes. Without being able to write a test for your fix, how will you convince a reviewer that it actually fixed what it fixed? All you can prove was your “simple” change didn’t break any behaviour that was covered by existing tests. </p>
<p>Fast forward to Friday and your one line change has finally been merged, along with a few new test cases to make sure it doesn’t happen again. Next week you’re going to sit down and try to figure out a bunch of weird crashes you caused while trying to stub out calls to the database. Eventually you gave up and copied some code from another test that uses a copy of production data. It’s much slower than it needs to be, but it worked, and was the only way to put a reasonable estimate on a job that had already consumed your entire week.</p>
<p>The moral of the story; was the test the reason it took nearly a week to make a one line fix?</p>
	</div>
	</article>
	<article>
	<div>
	
<p>The Go compiler’s SSA backend contains a facility to produce HTML debugging output of the compilation phases. This post covers how to print the SSA output for function <em>and</em> methods.</p>
<p>Let’s start with a sample program which contains a function, a value method, and a pointer method:</p>
<pre><code>package main
import (
 "fmt"
)
type Numbers struct {
 vals []int
}
func (n *Numbers) Add(v int) {
 n.vals = append(n.vals, v)
}
func (n Numbers) Average() float64 {
 sum := 0.0
 for _, num := range n.vals {
 sum += float64(num)
 }
 return sum / float64(len(n.vals))
}
func main() {
 var numbers Numbers
 numbers.Add(200)
 numbers.Add(43)
 numbers.Add(-6)
 fmt.Println(numbers.Average())
}</code></pre>
<p>Control of the SSA debugging output is via the <code>GOSSAFUNC</code> environment variable. This variable takes the name of the function to dump. This is <em>not</em> the functions fully qualified name. For <code>func main</code> above the name of the function is <code>main</code> <em>not</em> <code>main.main</code>.</p>
<pre>% <strong>env GOSSAFUNC=main go build</strong>
runtime
dumped SSA to ../../go/src/runtime/ssa.html
t
dumped SSA to ./ssa.html</pre>
<p>In this example <code>GOSSAFUNC=main</code> matched both <code>main.main</code> and a function called <code>runtime.main</code>.<a href="https://dave.cheney.net/2020/06/19/how-to-dump-the-gossafunc-graph-for-a-method#easy-footnote-bottom-1-4188"><sup>1</sup></a> This is a little unfortunate, but in practice probably not a big deal as, if you’re performance tuning your code, it won’t be in a giant spaghetti blob in <code>func main</code>. </p>
<p>What is more likely is your code will be in a <em>method</em>, so you’ve probably landed on this post looking for the correct incantation to dump the SSA output for a method.</p>
<p>To print the SSA debug for the pointer method <code>func (n *Numbers) Add</code>, the equivalent function name is<code>(*Numbers).Add</code>:<a href="https://dave.cheney.net/2020/06/19/how-to-dump-the-gossafunc-graph-for-a-method#easy-footnote-bottom-2-4188"><sup>2</sup></a></p>
<pre>% <strong>env "GOSSAFUNC=(*Numbers).Add" go build</strong>
t
dumped SSA to ./ssa.html</pre>
<p>To print the SSA debug for a value method <code>func (n Numbers) Average</code>, the equivalent function name is <code>(*Numbers).Average</code> <em>even though this is a value method</em>:</p>
<pre>% <strong>env "GOSSAFUNC=(*Numbers).Average" go build</strong>
t
dumped SSA to ./ssa.html</pre>
	</div>
	</article>
	<article>
	<div>
	
<p>Per the <a href="https://github.com/golang/proposal/blob/master/design/6977-overlapping-interfaces.md">overlapping interfaces proposal</a>, Go 1.14 now permits embedding of interfaces with overlapping method sets. This is a brief post explain what this change means:</p>
<p>Let’s start with the definition of the three key interfaces from the <code>io</code> package; <code>io.Reader</code>, <code>io.Writer</code>, and <code>io.Closer</code>:</p>
<pre><code>package io
type Reader interface {
 Read([]byte) (int, error)
}
type Writer interface {
 Write([]byte) (int, error)
}
type Closer interface {
 Close() error
} </code></pre>
<p>Just as embedding a type inside a struct allows the embedded type’s fields and methods to be accessed as if it were declared on the embedding type<a href="https://dave.cheney.net/2020/05/24/diamond-interface-composition-in-go-1-14#easy-footnote-bottom-1-4179"><sup>1</sup></a>, the process is true for interfaces. Thus there is no difference between explicitly declaring</p>
<pre><code>type ReadCloser interface {
 Read([]byte) (int, error)
 Close() error
}</code></pre>
<p>and using embedding to compose the interface</p>
<pre><code>type ReadCloser interface {
 Reader
 Closer
}</code></pre>
<p>You can even mix and match </p>
<pre><code>type WriteCloser interface {
 Write([]byte) (int, error)
 Closer
}</code></pre>
<p>However, prior to Go 1.14, if you continued to compose interface declarations in this manner you would likely find that something like this,</p>
<pre><code>type ReadWriteCloser interface {
 ReadCloser
 WriterCloser
}</code></pre>
<p>would fail to compile</p>
<pre>% <strong>go build interfaces.go</strong>
command-line-arguments
./interfaces.go:27:2: duplicate method Close</pre>
<p>Fortunately, with Go 1.14 this is no longer a limitation, thus solving problems that typically occur with diamond-shaped embedding graphs.</p>
<p>However, there is a catch that I ran into attempting to demonstrate this feature to the local user group–this feature is only enabled when the Go compiler uses the 1.14 (or later) spec.</p>
<p>As near as I can make out the rules for which version of the Go spec is used during compilation appear to be:</p>
<ol><li>If your source code is stored inside <code>GOPATH</code> (or you have <em>disabled</em> modules with <code>GO111MODULE=off</code>) then the version of the Go spec used to compile with matches the version of the compiler you are using. Said another way, if you have Go 1.13 installed, your Go version is 1.13. If you have Go 1.14 installed, your version is 1.14. No surprises here.</li><li>If your source code is stored outside <code>GOPATH</code> (or you have forced modules on with <code>GO111MODULE=on</code>) then the <code>go</code> tool will take the Go version from the <code>go.mod</code> file.</li><li> If there is no Go version listed in <code>go.mod</code> then the version of the spec will be the version of Go installed. This is identical to point 1.</li><li>If you are in module mode, either by being outside <code>GOPATH</code> or with <code>GO111MODULE=on</code>, but there is no <code>go.mod</code> file in the current, or any parent, directory then the version of the Go spec used to compile your code defaults to Go 1.13.</li></ol>
<p>The last point caught me out.</p>
	</div>
	</article>
	<article>
	<div>
	
<p>A few days ago Fatih posted <a href="https://twitter.com/fatih/status/1260683136842608640">this question</a> on twitter.</p>
<p>I’m going to attempt to give my answer, however to do that I need to apply some simplifications as my previous attempts to answer it involved a lot of phrases like <em>a pointer to a pointer</em>, and other unhelpful waffling. Hopefully my simplified answer can be useful in building a mental framework to answer Fatih’s original question.</p>
<h2>Restating the question</h2>
<p>Fatih’s original tweet showed <a href="https://play.golang.org/p/g2yUIYrV67F">four different variations</a> of <code>json.Unmarshal</code>. I’m going to focus on the last two, which I’ll rewrite a little:</p>
<pre><code>package main
import (
 "encoding/json"
 "fmt"
)
type Result struct {
 Foo string `json:"foo"`
}
func main() {
 content := []byte(`{"foo": "bar"}`)
 var result1, result2 *Result
 err := json.Unmarshal(content, &amp;result1)
 fmt.Println(result1, err) // &amp;{bar} &lt;nil&gt;
 err = json.Unmarshal(content, result2)
 fmt.Println(result2, err) // &lt;nil&gt; json: Unmarshal(nil *main.Result)
}</code></pre>
<p>Restated in words, <code>result1</code> and <code>result2</code> are the same type; <code>*Result</code>. Decoding into <code>result1</code> works as expected, whereas decoding into <code>result2</code> causes the <code>json</code> package to complain that the value passed to <code>Unmarshal</code> is <code>nil</code>. However, both values were declared without an initialiser so both would have taken on the type’s zero value, <code>nil</code>.</p>
<p>Eagle eyed readers will have spotted that the reason for the difference is the first<code> </code>invocation is passed <code>&amp;result1</code>, while the second is passed <code>result2</code>, but this explanation is unsatisfactory because the documentation for <code>json.Unmarshal</code> states:</p>
<blockquote><p>Unmarshal parses the JSON-encoded data and stores the result in the value pointed to by v. <strong>If v is nil or not a pointer</strong>, Unmarshal returns an InvalidUnmarshalError.</p></blockquote>
<p>Which is confusing because <code>result1</code> and <code>result2</code> <em>are</em> pointers. Furthermore, without initialisation, both <em>are</em> <code>nil</code>. Now, the documentation is correct (as you’d expect from a package that has been hammered on for a decade), but explaining <em>why</em> takes a little more investigation.</p>
<h2>Functions receive a copy of their arguments</h2>
<p>Every assignment in Go is a copy, this includes function arguments and return values.</p>
<pre><code>package main
import (
 "fmt"
)
func increment(v int) {
 v++
}
func main() {
 v := 1
 increment(v)
 fmt.Println(v) // 1
}</code></pre>
<p>In this example, <code>increment</code> is operating on a <em>copy</em> of <code>main</code>‘s <code>v</code>. This is because the <code>v</code> declared in <code>main</code> and <code>increment</code>‘s <code>v</code> parameter have different addresses in memory. Thus changes to <code>increment</code>‘s <code>v</code> cannot affect the contents of <code>main</code>‘s <code>v</code>.</p>
<pre><code>package main
import (
 "fmt"
)
func increment(v *int) {
 *v++
}
func main() {
 v := 1
 increment(&amp;v)
 fmt.Println(v) // 2
}</code></pre>
<p>If we wanted to write <code>increment</code> in a way that it could affect the contents of its caller we would need to pass a reference, a pointer, to <code>main.v</code>.<a href="https://dave.cheney.net/2020/05/16/fatihs-question#easy-footnote-bottom-1-4153"><sup>1</sup></a> This example demonstrates why <code>json.Unmarshal</code> needs a pointer to the value to decode JSON into.</p>
<h2>Pointers to pointers</h2>
<p>Returning to the original question, both <code>result1</code> and <code>result2</code> are declared as <code>*Result</code>, that is, pointers to a <code>Result</code> value. We established that you have to pass the address of caller’s value to <code>json.Unmarshal</code> otherwise it won’t be able to alter the contents of the caller’s value. Why then must we pass the address of <code>result1</code>, a <code>**Result</code>, a pointer to a pointer to a <code>Result</code>, for the operation to succeed.</p>
<p>To explain this another detour is required. Consider this code:</p>
<pre><code>package main
import (
 "encoding/json"
 "fmt"
)
type Result struct {
 Foo *string `json:"foo"`
}
func main() {
 content := []byte(`{"foo": "bar"}`)
 var result1 *Result
 err := json.Unmarshal(content, &amp;result1)
 fmt.Printf("%#v %v", result1, err) // &amp;main.Result{Foo:(*string)(0xc0000102f0)} &lt;nil&gt;
}</code></pre>
<p>In this example <code>Result</code> contains a pointer typed field, <code>Foo *string</code>. During JSON decoding <code>Unmarshal</code> allocated a new <code>string</code> value, stored the value <code>bar</code> in it, then placed the address of the string in <code>Result.Foo</code>. This behaviour is quite handy as it frees the caller from having to initialise <code>Result.Foo</code> and makes it easier to detect when a field was not initialised because the JSON did not contain a value. Beyond the convenience this offers for simple examples it would be prohibitively difficult for the caller to properly initialise all the reference type fields in a structure before decoding unknown JSON without first inspecting the incoming JSON which itself may be problematic if the input is coming from an <code>io.Reader</code> without the ability to rewind the input.</p>
<blockquote><p>To unmarshal JSON into a pointer, Unmarshal first handles the case of the JSON being the JSON literal null. In that case, Unmarshal sets the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the value pointed at by the pointer. <strong>If the pointer is nil, Unmarshal allocates a new value for it to point to</strong>.</p></blockquote>
<p><code>json.Unmarshal</code>‘s handling of pointer fields is clearly documented, and works as you would expect, allocating a new value whenever there is a need to decode into a pointer shaped field. It is this behaviour that gives us a hint to what is happening in the original example.</p>
<p>We’ve seen that when <code>json.Unmarshal</code> encounters a field which points to <code>nil</code> it will allocate a new value of the correct type and assign its address the field before proceeding. Not only is does behaviour is applied recursively–for example in the case of a complex structure which contains pointers to other structures–but it also applies to the <em>value passed to <code>Unmarshal</code>.</em></p>
<pre><code>package main
import (
 "encoding/json"
 "fmt"
)
func main() {
 content := []byte(`1`)
 var result *int
 err := json.Unmarshal(content, &amp;result)
 fmt.Println(*result, err) // 1 &lt;nil&gt;
}</code></pre>
<p>In this example <code>result</code> is not a struct, but a simple <code>*int</code> which, lacking an initialiser, defaults to <code>nil</code>. After JSON decoding, <code>result</code> now points to an <code>int</code> with the value <code>1</code>.</p>
<h2>Putting the pieces together</h2>
<p>Now I think I’m ready to take a shot at answering Fatih’s question.</p>
<p><code>json.Unmarshal</code> requires the address of the variable you want to decode into, otherwise it would decode into a temporary copy which would be discard on return. Normally this is done by declaring a value, then passing its address, or explicitly initialising the the value</p>
<pre><code>var result1 Result
err := json.Unmarshal(content, &amp;result1) // this is fine
var result2 = new(Result)
err = json.Unmarshal(content, result2) // and this
var result3 = &amp;Result{}
err = json.Unmarshal(content, result3) // this is also fine</code></pre>
<p>In all three cases the address that the <code>*Result</code> points too is not <code>nil</code>, it points to initialised memory that <code>json.Unmarshal</code> decodes into.</p>
<p>Now consider what happens when <code>json.Unmarshal</code> encounters this</p>
<pre><code>var result4 *Result
err = json.Unmarshal(content, result4) // err json: Unmarshal(nil *main.Result)</code></pre>
<p><code>result2</code>, <code>result3</code>, and the expression <code>&amp;result1</code> point to a <code>Result</code>. However <code>result4</code>, even though it has the same type as the previous three, does not point to initialised memory, it points to <code>nil</code>. Thus, according to the examples we saw previously, before <code>json.Unmarshal</code> can decode into it, the memory <code>result4</code> points too must be initialised. </p>
<p>However, because each function receives a copy of its arguments, the caller’s <code>result4</code> variable and the copy inside <code>json.Unmarshal</code> are unique. <code>json.Unmarshal</code> can allocate a new <code>Result</code> value and decode into it, but it cannot alter <code>result4</code> to point to this new value because it was not provided with a reference to <code>result4</code>, only a copy of its contents.</p>
	</div>
	</article>
	<article>
	<div>
	
<p>Conventional wisdom dictates that the larger the number of types declared in a Go program, the larger the resulting binary. Intuitively this makes sense, after all, what’s the point in defining a bunch of types if you’re not going to write code that operates on them. However, part of the job of a linker is to detect functions which are not referenced by a program–say they are part of a library of which only a subset of functionality is used–and remove them from the final output. Yet, the adage mo’ types, mo’ binary holds true for the majority of Go programs.</p>
<p>In this post I’ll dig into what equality, in the context of a Go program, means and why changes <a href="https://github.com/golang/net/commit/e0ff5e5a1de5b859e2d48a2830d7933b3ab5b75f">like this</a> have a measurable impact on the size of a Go program.</p>
<h2>Defining equality between two values</h2>
<p>The Go spec defines the concepts of assignability and equality. Assignabiity is the act of assigning a value to an identifier. Not everything which is declared can be assigned, for example constants and functions. Equality is the act of comparing two identifies by asking <em>are their contents the same?</em></p>
<p>Being a strongly typed language, the notion of sameness is fundamentally rooted in the identifier’s type. Two things can only be the same if they are of the same type. Beyond that, the type of the values defines how they are compared.</p>
<p>For example, integers are compared arithmetically. For pointer types, equality is determining if the addresses they point too are the same. Reference types like maps and channels, like pointers, are considered to be the same if they have the same address.</p>
<p>These are all examples of bitwise equality, that is, if the bit patterns of the memory that value occupies are the same, those values are equal. This is known as memcmp, short for memory comparison, as equality is defined by comparing the contents of two areas of memory.</p>
<p>Hold on to this idea, I’ll come back to in a second.</p>
<h2>Struct equality</h2>
<p>Beyond scalar types like integers, floats, and pointers is the realm of compound types; structs. All structs are laid out in memory in program order, thus this declaration:</p>
<pre><code>type S struct {
 a, b, c, d int64
}</code></pre>
<p>will consume 32 bytes of memory; 8 bytes for <code>a</code>, then 8 bytes for <code>b</code>, and so on. The spec says that <em>struct values are comparable if all their fields are comparable</em>. Thus two structs are equal iff each of their fields are equal.</p>
<pre><code>a := S{1, 2, 3, 4}
b := S{1, 2, 3, 4}
fmt.Println(a == b) // prints true</code></pre>
<p>Under the hood the compiler uses memcmp to compare the 32 bytes of <code>a</code> and <code>b</code>.</p>
<h2>Padding and alignment</h2>
<p>However the simplistic bitwise comparison strategy will fail in situations like this:</p>
<pre><code>type S struct {
 a byte
 b uint64
 c int16
 d uint32
}
func main()
 a := S{1, 2, 3, 4}
 b := S{1, 2, 3, 4}
 fmt.Println(a == b) // prints true
}</code></pre>
<p>The code compiles, the comparison is still true, but under the hood the compiler cannot rely on comparing the bit patterns of <code>a</code> and <code>b</code> because the structure contains <em>padding</em>.</p>
<p>Go requires each field in a struct to be naturally aligned. 2 byte values must start on an even address, four byte values on an address divisible by 4, and so on<a href="https://dave.cheney.net/2020/05/09/ensmallening-go-binaries-by-prohibiting-comparisons#easy-footnote-bottom-1-4116"><sup>1</sup></a>. The compiler inserts padding to ensure the fields are <em>aligned</em> to according to their type and the underlying platform. In effect, after padding, this is what the compiler sees<a href="https://dave.cheney.net/2020/05/09/ensmallening-go-binaries-by-prohibiting-comparisons#easy-footnote-bottom-2-4116"><sup>2</sup></a>:</p>
<pre><code>type S struct {
 a byte
 _ [7]byte // padding
 b uint64
 c int16
 _ [2]int16 // padding
 d uint32
}</code></pre>
<p>Padding exists to ensure the correct field alignments, and while it does take up space in memory, the contents of those padding bytes are unknown. You might assume that, being Go, the padding bytes are always zero, but it turns out that’s not the case–the contents of padding bytes are simply not defined. Because they’re not defined to always be a certain value, doing a bitwise comparison may return false because the nine bytes of padding spread throughout the 24 bytes of <code>S</code> are may not be the same.</p>
<p>The Go compiler solves this problem by generating what is known as an equality function. In this case <code>S</code>‘s equality function knows how to compare two values of type <code>S</code> by comparing only the fields in the function while skipping over the padding.</p>
<h2>Type algorithms</h2>
<p>Phew, that was a lot of setup to illustrate why, for each type defined in a Go program, the compiler may generate several supporting functions, known inside the compiler as the type’s algorithms. In addition to the equality function the compiler will generate a hash function if the type is used as a map key. Like the equality function, the hash function must consider factors like padding when computing its result to ensure it remains stable. </p>
<p>It turns out that it can be hard, and sometimes non obvious, to intuit when the compiler will generate these functions–it’s more than you’d expect–and it can be hard for the linker to eliminate the ones that are not needed as reflection often causes the linker to be more conservative when trimming types.</p>
<h2>Reducing binary size by prohibiting comparisons</h2>
<p>Now we’re at a point to explain Brad’s change. By adding an incomparable field <a href="https://dave.cheney.net/2020/05/09/ensmallening-go-binaries-by-prohibiting-comparisons#easy-footnote-bottom-3-4116"><sup>3</sup></a> to the type, the resulting struct is by extension incomparable, thus forcing the compiler to elide the generation of eq and hash algorithms, short circuiting the linkers elimination of those types and, in practice, reducing the size of the final binary. As an example of this technique, this program:</p>
<pre><code>package main
import "fmt"
func main() {
 type t struct {
 // _ [0][]byte uncomment to prevent comparison
 a byte
 b uint16
 c int32
 d uint64
 }
 var a t
 fmt.Println(a)
}</code></pre>
<p>when compiled with Go 1.14.2 (darwin/amd64), decreased from 2174088 to 2174056, a saving of 32 bytes. In isolation this 32 byte saving may seem like small beer, but consider that equality and hash functions can be generated for every type in the transitive closure of your program and all its dependencies, and the size of these functions varies depending on the size of the type and its complexity, prohibiting them can have a sizeable impact on the final binary over and above the old saw of <code>-ldflags="-s -w"</code>.</p>
<p>The bottom line, if you don’t wish to make your types comparable, a hack like this enforces it at the source level while contributing to a small reduction in the size of your binary.</p>
<hr />
<p>Addendum: thanks to Brad’s prodding, Go 1.15 already has a bunch of improvements by <a href="https://go-review.googlesource.com/c/go/+/231397">Cherry Zhang</a> and <a href="https://go-review.googlesource.com/c/go/+/191198">Keith Randall</a> that fix the most egregious of the failures to eliminate unnecessary equality and hash functions (although I suspect it was also to avoid the proliferation of this class of CLs).</p>
	</div>
	</article>
	<article>
	<div>
	
<p>In the <a href="https://dave.cheney.net/2020/04/25/inlining-optimisations-in-go">previous post</a> I discussed how leaf inlining allows the Go compiler to reduce the overhead of function calls and extend optimisation opportunities across function boundaries. In this post I’ll discuss the limits of inlining and leaf vs mid-stack inlining.</p>
<h2>The limits of inlining</h2>
<p>Inlining a function into its caller removes the call’s overhead and increases the opportunity for the compiler to apply additional optimisations so the question should be asked, if some inlining is good, would more be better, <em>why not inline as much as possible?</em></p>
<p>Inlining trades possibly larger program sizes for potentially faster execution time. The main reason to limit inlining is creating many inlined copies of a function can increase compile time and result in larger binaries for marginal gain. Even taking into account the opportunities for further optimisation, aggressive inlining tends to increase the size of, and the time too compile, the resulting binary.</p>
<p>Inlining works best for <a href="https://medium.com/@joshsaintjacque/small-functions-considered-awesome-c95b3fd1812f">small functions</a> that do relatively little work compared to the overhead of calling them. As the size of a function grows, the time saved avoiding the call’s overhead diminishes relative to the work done inside the function. Larger functions tend to be more complex, thus the benefits of optimising their inlined forms vs in situ are reduced.</p>
<h2>Inlining budget</h2>
<p>During compilation each function’s inlineabilty is calculated using what is known as the <em>inlining budget</em><a href="https://dave.cheney.net/2020/05/02/mid-stack-inlining-in-go#easy-footnote-bottom-1-4076"><sup>1</sup></a>. The cost calculation can be tricky to internalise but is broadly one unit per node in the AST for simple things like unary and binary operations but can be higher for complex operations like <code>make</code>. Consider this example:</p>
<pre><code>package main
func small() string {
 s := "hello, " + "world!"
 return s
}
func large() string {
 s := "a"
 s += "b"
 s += "c"
 s += "d"
 s += "e"
 s += "f"
 s += "g"
 s += "h"
 s += "i"
 s += "j"
 s += "k"
 s += "l"
 s += "m"
 s += "n"
 s += "o"
 s += "p"
 s += "q"
 s += "r"
 s += "s"
 s += "t"
 s += "u"
 s += "v"
 s += "w"
 s += "x"
 s += "y"
 s += "z"
 return s
}
func main() {
 small()
 large()
}</code></pre>
<p>Compiling this function with <code>-gcflags=-m=2</code> allows us to see the cost the compiler assigns to each function.</p>
<pre>% <strong>go build -gcflags=-m=2 inl.go </strong>
# command-line-arguments
./inl.go:3:6: can inline small with cost 7 as: func() string { s := "hello, world!"; return s }
./inl.go:8:6: cannot inline large: function too complex: cost 82 exceeds budget 80
./inl.go:38:6: can inline main with cost 68 as: func() { small(); large() }
./inl.go:39:7: inlining call to small func() string { s := "hello, world!"; return s }</pre>
<p>The compiler determined that <code>func small()</code> can be inlined due to its cost of 7. <code>func large()</code> was determined to be too expensive. <code>func main()</code>has been marked as eligible and assigned a cost of 68; 7 from the body of <code>small</code>, 57 from the function call to <code>small</code> and the remainder in its own overhead.</p>
<p>The inlining budget can be controlled to some degree with the <code>-gcflag=-l</code> flag. Currently the values that apply are:</p>
<ul><li><code>-gcflags=-l=0</code> is the default level of inlining.</li><li><code>-gcflags=-l</code> (or <code>-gcflags=-l=1</code>) disables inlining.</li><li><code>-gcflags=-l=2</code> and <code>-gcflags=-l=3</code> are currently unused and have no effect over <code>-gcflags=-l=0</code></li><li><code>-gcflags=-l=4</code> reduces the cost for inlining non-leaf functions and calls through interfaces.<a href="https://dave.cheney.net/2020/05/02/mid-stack-inlining-in-go#easy-footnote-bottom-2-4076"><sup>2</sup></a></li></ul>
<h3>Hairy optimisations</h3>
<p>Some functions with a relatively low inlining cost may be ineligible because of their complexity. This is known as the function’s hairiness as the semantics of some operations are hard to reason about once inlined, for example <code>recover</code>, <code>break</code>. Others, like <code>select</code> and <code>go</code>, involve co-ordination with the runtime so the extra effort of inlining doesn’t pay for itself.</p>
<p>The list of hairy statements also includes things like <code>for </code>and <code>range</code> which don’t have an inherently large cost, but simply haven’t been optimised yet.</p>
<h2>Mid stack inlining</h2>
<p>Historically the Go compiler only performed leaf inlining–only functions which did not call other functions were eligible. In the context of the hairiness discussion previously, a function call would disqualify the function from being inlined.</p>
<p>Enter mid stack inlining which, as its name implies, allows functions in the middle of a call stack to be inlined without requiring everything below them to be eligible. Mid stack inlining was introduced by David Lazar in Go 1.9 and improved in subsequent releases. <a href="https://docs.google.com/presentation/d/1Wcblp3jpfeKwA0Y4FOmj63PW52M_qmNqlQkNaLj0P5o/edit#slide=id.p">This presentation</a> goes into some of the difficulties with retaining the behaviour of stack traces and <code>runtime.Callers</code> in code paths that had been heavily inlined.</p>
<p>We see an example of mid-stack inlining in the previous example. After inlining, <code>func main()</code> contains the body of <code>func small()</code> and a call to <code>func large()</code>, thus it is considered a non-leaf function. Historically this would have prevented it from being further inlined even though its combined cost was less than the inlining budget.</p>
<p>The primary use case for mid stack inlining is to reduce the overhead of a path through the call stack. Consider this example:</p>
<pre><code>package main
import (
 "fmt"
 "strconv"
)
type Rectangle struct {}
//go:noinline
func (r *Rectangle) Height() int {
 h, _ := strconv.ParseInt("7", 10, 0)
 return int(h)
}
func (r *Rectangle) Width() int {
 return 6
}
func (r *Rectangle) Area() int { return r.Height() * r.Width() }
func main() {
 var r Rectangle
 fmt.Println(r.Area())
}</code></pre>
<p>In this example <code>r.Area()</code> is a simple function which calls two others. <code>r.Width()</code> can be inlined while <code>r.Height()</code>, simulated here with the <code>//go:noinline</code> annotation, cannot. <a href="https://dave.cheney.net/2020/05/02/mid-stack-inlining-in-go#easy-footnote-bottom-3-4076"><sup>3</sup></a></p>
<pre>% <strong>go build -gcflags='-m=2' square.go </strong>
# command-line-arguments
./square.go:12:6: cannot inline (*Rectangle).Height: marked go:noinline 
./square.go:17:6: can inline (*Rectangle).Width with cost 2 as: method(*Rectangle) func() int { return 6 }
./square.go:21:6: <strong>can inline (*Rectangle).Area with cost 67 as: method(*Rectangle) func() int { return r.Height() * r.Width() } </strong>./square.go:21:61: inlining call to (*Rectangle).Width method(*Rectangle) func() int { return 6 } 
./square.go:23:6: cannot inline main: function too complex: cost 150 exceeds budget 80 
./square.go:25:20: inlining call to (*Rectangle).Area method(*Rectangle) func() int { return r.Height() * r.Width() }
./square.go:25:20: inlining call to (*Rectangle).Width method(*Rectangle) func() int { return 6 }</pre>
<p>As the multiplication performed by <code>r.Area()</code> is cheap compared to the overhead of calling it, inlining <code>r.Area()</code>‘s single expression is a net win even if its downstream caller to <code>r.Height()</code> remains ineligible.</p>
<h3>Fast path inlining</h3>
<p>The most startling example of the power of mid-stack inlining comes from 2019 when <a href="https://go-review.googlesource.com/c/go/+/148959">Carlo Alberto Ferraris improved the performance</a> of <code>sync.Mutex.Lock()</code> by allowing the fast path of the lock–the uncontended case–to be inlined into its caller. Prior to this change <code>sync.Mutex.Lock()</code> was a large function containing many hairy conditions which made it ineligible to be inlined. Even in the case where the lock was available, the caller had to pay the overhead of calling <code>sync.Mutex.Lock()</code>.</p>
<p>Carlo’s change split <code>sync.Mutex.Lock()</code> into two functions (a process he dubbed <em>outlining</em>). The outer <code>sync.Mutex.Lock()</code> method now calls <code>sync/atomic.CompareAndSwapInt32()</code> and returns to the caller immediately if the CAS succeeds. If not, the function falls through to <code>sync.Mutex.lockSlow()</code> which handles the slow path required to register interest on the lock and park the goroutine.<a href="https://dave.cheney.net/2020/05/02/mid-stack-inlining-in-go#easy-footnote-bottom-4-4076"><sup>4</sup></a></p>
<pre>% <strong>go build -gcflags='-m=2 -l=0' sync 2&gt;&amp;1 | grep '(*Mutex).Lock'</strong>
../go/src/sync/mutex.go:72:6: can inline (*Mutex).Lock with cost 69 as: method(*Mutex) func() { if "sync/atomic".CompareAndSwapInt32(&amp;m.state, 0, mutexLocked) { if race.Enabled {  }; return  }; m.lockSlow() }</pre>
<p>By splitting the function into an easily inlineable outer function, falling through to a complex inner function to handle the slow path Carlo’s combined mid stack inlining and the <a href="https://dave.cheney.net/2019/08/20/go-compiler-intrinsics">compiler’s support for intrinsic operations</a> to reduce the cost of an uncontended lock by 14%. Then he repeated the trick for an additional 9% saving in <code><a href="https://go-review.googlesource.com/c/go/+/152698">sync.RWMutex.Unlock()</a></code>.</p>
	</div>
	</article>
	<article>
	<div>
	
<p>This is a post about how the Go compiler implements inlining and how this optimisation affects your Go code. </p>
<p><em>n.b.</em> This article focuses on <em>gc</em>, the de facto Go compiler from <a href="https://github.com/golang/go">golang.org</a>. The concepts discussed apply broadly to other Go compilers like gccgo and tinygo but may differ in implementation and efficacy.</p>
<h2>What is inlining?</h2>
<p>Inlining is the act of combining smaller functions into their respective callers. In the early days of computing this optimisation was typically performed by hand. Nowadays inlining is one of a class of fundamental optimisations performed automatically during the compilation process.</p>
<h2>Why is inlining important?</h2>
<p>Inlining is important for two reasons. The first is it removes the overhead of the function call itself. The second is it permits the compiler to more effectively apply other optimisation strategies.</p>
<h3>Function call overhead</h3>
<p>Calling a function<a href="https://dave.cheney.net/2020/04/25/inlining-optimisations-in-go#easy-footnote-bottom-1-4053"><sup>1</sup></a> in any language carries a cost. There are the overheads of marshalling parameters into registers or onto the stack (depending on the ABI) and reversing the process on return. Invoking a function call involves jumping the program counter from one point in the instruction stream to another which can cause a pipeline stall. Once inside the function there is usually some preamble required to prepare a new stack frame for the function to execute and a similar epilogue needed to retire the frame before returning to the caller.</p>
<p>In Go a function call carries additional costs to support dynamic stack growth. On entry the amount of stack space available to the goroutine is compared to the amount required for the function. If insufficient stack space is available, the preamble jumps into the runtime logic that grows the stack by copying it to a new, larger, location. Once this is done the runtime jumps back to the start of the original function, the stack check is performed again, which now passes, and the call continues. In this way goroutines can start with a small stack allocation which grows only when needed.<a href="https://dave.cheney.net/2020/04/25/inlining-optimisations-in-go#easy-footnote-bottom-2-4053"><sup>2</sup></a></p>
<p>This check is cheap–only a few instructions–and because goroutine stacks grows geometrically the check rarely fails. Thus, the branch prediction unit inside a modern processor can hide the cost of the stack check by assuming it will always be successful. In the case where the processor mis-predicts the stack check and has to discard the work done while it was executing speculatively, the cost of the pipeline stall is relatively small compared to the cost of the work needed for the runtime to grow a goroutines stack.</p>
<p>While the overhead of the generic and Go specific components of each function call are well optimised by modern processors using speculative execution techniques, those overheads cannot be entirely eliminated, thus each function call carries with it a performance cost over and above the time it takes to perform useful work. As a function call’s overhead is fixed, smaller functions pay a larger cost relative to larger ones because they tend to do less useful work per invocation.</p>
<p>The solution to eliminating these overheads must therefore be to eliminate the function call itself, which the Go compiler does, under certain conditions, by replacing the call to a function with the contents of the function. This is known as <em>inlining</em> because it brings the body of the function in line with its caller.</p>
<h3>Improved optimisation opportunities</h3>
<p>Dr. Cliff Click describes inlining as <em>the</em> optimisation performed by modern compilers as it forms the basis for optimisations like constant propagation and dead code elimination. In effect, inlining allows the compiler to <em>see </em><i>furthe</i>r, allowing it to observe, in the context that a particular function is being called, logic that can be further simplified or eliminated entirely. As inlining can be applied recursively optimisation decisions can be made not only in the context of each individual function, but also applied to the chain of functions in a call path.</p>
<h2>Inlining in action</h2>
<p>The effects of inlining can be demonstrated with this small example</p>
<pre><code>package main
import "testing"
//go:noinline
func max(a, b int) int {
 if a &gt; b {
 return a
 }
 return b
}
var Result int
func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i &lt; b.N; i++ {
 r = max(-1, i)
 }
 Result = r
}</code></pre>
<p>Running this benchmark gives the following result:<a href="https://dave.cheney.net/2020/04/25/inlining-optimisations-in-go#easy-footnote-bottom-3-4053"><sup>3</sup></a></p>
<pre>% <strong>go test -bench=. </strong>
BenchmarkMax-4   530687617         2.24 ns/op</pre>
<p>The cost of <code>max(-1, i)</code> is around 2.24 nanoseconds on my 2015 MacBook Air. Now let’s remove the <code>//go:noinline</code> pragma and see the result:</p>
<pre>% <strong>go test -bench=. </strong>
BenchmarkMax-4   1000000000         0.514 ns/op</pre>
<p>From 2.24 ns to 0.51 ns, or according to <code>benchstat</code>, a 78% improvement.</p>
<pre>% <strong>benchstat {old,new}.txt</strong>
name   old time/op  new time/op  delta
Max-4  2.21ns ± 1%  0.49ns ± 6%  -77.96%  (p=0.000 n=18+19)</pre>
<p>Where did these improvements come from? </p>
<p>First, the removal of the function call and associated preamble<a href="https://dave.cheney.net/2020/04/25/inlining-optimisations-in-go#easy-footnote-bottom-4-4053"><sup>4</sup></a> was a major contributor. Pulling the contents of <code>max</code> into its caller reduced the number of instructions executed by the processor and eliminated several branches.</p>
<p>Now the contents of <code>max</code> are visible to the compiler as it optimises <code>BenchmarkMax</code> it can make some additional improvements. Consider that once <code>max</code> is inlined, this is what the body of <code>BenchmarkMax</code> looks like to the compiler:</p>
<pre><code>func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i &lt; b.N; i++ {
 if -1 &gt; i {
 r = -1
 } else {
 r = i
 }
 }
 Result = r
}</code></pre>
<p>Running the benchmark again we see our manually inlined version performs as well as the version inlined by the compiler</p>
<pre>% <strong>benchstat {old,new}.txt</strong>
name   old time/op  new time/op  delta
Max-4  2.21ns ± 1%  0.48ns ± 3%  -78.14%  (p=0.000 n=18+18)</pre>
<p>Now the compiler has access to the result of inlining <code>max</code> into <code>BenchmarkMax</code> it can apply optimisation passes which were not possible before. For example, the compiler has noted that <code>i</code> is initialised to <code>0</code> and only incremented so any comparison with <code>i</code> can assume <code>i</code> will never be negative. Thus, the condition <code>-1 &gt; i</code> will never be true.<a href="https://dave.cheney.net/2020/04/25/inlining-optimisations-in-go#easy-footnote-bottom-5-4053"><sup>5</sup></a></p>
<p>Having proved that <code>-1 &gt; i</code> will never be true, the compiler can simplify the code to</p>
<pre><code>func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i &lt; b.N; i++ {
 if false {
 r = -1
 } else {
 r = i
 }
 }
 Result = r
}</code></pre>
<p>and because the branch is now a constant, the compiler can eliminate the unreachable path leaving it with</p>
<pre><code>func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i &lt; b.N; i++ {
 r = i
 }
 Result = r
}</code></pre>
<p>Thus, through inlining and the optimisations it unlocks, the compiler has reduced the expression <code>r = max(-1, i)</code> to simply <code>r = i</code>.</p>
<h2>The limits of inlining</h2>
<p>In this article I’ve discussed, so called, <em>leaf</em> inlining; the act of inlining a function at the bottom of a call stack into its direct caller. Inlining is a recursive process, once a function has been inlined into its caller, the compiler may inline the resulting code into <em>its</em> caller, as so on. For example, this code </p>
<pre><code>func BenchmarkMaxMaxMax(b *testing.B) {
 var r int
 for i := 0; i &lt; b.N; i++ {
 r = max(max(-1, i), max(0, i))
 }
 Result = r
}</code></pre>
<p>runs as fast as the previous example as the compiler is able to repeatedly apply the optimisations outlined above to reduce the code to the same <code>r = i</code> expression.</p>
<p>In the next article I’ll discuss an alternative inlining strategy when the Go compiler wishes to inline a function in the middle of a call stack. Finally I’ll discuss the limits that the compiler is prepared to go to to inline code, and which Go constructs are currently beyond its capability.</p>
	</div>
	</article>
	<article>
	<div>
	
<p>The <code>testing</code> package is one of my favourite packages in the Go standard library, not just because of its low noise approach to unit testing, but, over the lifetime of Go, it has received a steady stream of quality of life improvements driven by real world usage.</p>
<p>The most recent example of this is, in Go 1.14, <code>go test -v</code> will stream <code>t.Log</code> output as it happens, rather than hoarding it til the end of the test run. Here’s an example;</p>
<pre><code>package main
import (
	"fmt"
	"testing"
	"time"
)
func TestLogStreaming(t *testing.T) {
	for i := 0; i &lt; 5; i++ {
	time.Sleep(300 * time.Millisecond)
	fmt.Println("fmt.Println:", i)
	t.Log("t.Log:", i)
	}
}</code></pre>
<p><em>Note:</em> Calling <code>fmt.Println</code> inside a test is generally considered a no no as it bypasses the <code>testing</code> package’s output buffering irrespective of the <code>-v</code> flag. However, for this example, it‘s necessary to demonstrate the streaming <code>t.Log</code> change.</p>
<pre><code>% go1.13 test -v tlog_test.go
=== RUN TestLogStreaming
fmt.Println: 0
fmt.Println: 1
fmt.Println: 2
fmt.Println: 3
fmt.Println: 4
--- PASS: TestLogStreaming (1.52s)
 tlog_test.go:13: t.Log: 0
 tlog_test.go:13: t.Log: 1
 tlog_test.go:13: t.Log: 2
 tlog_test.go:13: t.Log: 3
 tlog_test.go:13: t.Log: 4
PASS
ok command-line-arguments 1.971s</code></pre>
<p>Under Go 1.13 and earlier the <code>fmt.Println</code> lines output immediately. <code>t.Log</code> lines are buffered and are printed after the test completes.</p>
<pre><code>% go1.14 test -v tlog_test.go
=== RUN TestLogStreaming
fmt.Println: 0
 TestLogStreaming: tlog_test.go:13: t.Log: 0
fmt.Println: 1
 TestLogStreaming: tlog_test.go:13: t.Log: 1
fmt.Println: 2
 TestLogStreaming: tlog_test.go:13: t.Log: 2
fmt.Println: 3
 TestLogStreaming: tlog_test.go:13: t.Log: 3
fmt.Println: 4
 TestLogStreaming: tlog_test.go:13: t.Log: 4
--- PASS: TestLogStreaming (1.51s)
PASS
ok command-line-arguments 1.809s</code></pre>
<p>Under Go 1.14 the <code>fmt.Println</code> and <code>t.Log</code> lines are interleaved, rather than waiting for the test to complete, demonstrating that test output is streamed when <code>go test -v</code> is used.</p>
<p>This is a great quality of life improvement for integration style tests that often retry for long periods when the test is failing. Streaming <code>t.Log</code> output will help Gophers debug those test failures without having to wait until the entire test times out to receive their output. </p>
	</div>
	</article>
	<article>
	<div>
	
<p>Programmers have a tendency to be superstitious. Particularly, when a programmer hears that copies are expensive, they start to see them everywhere, especially when they learn that, in Go, <em>every</em> assignment is a copy.</p>
<p>Consider this code; <code>x</code> is three orders of magnitude larger than <code>y</code>, is the assignment of <code>x</code> to <code>a</code> more expensive than the assignment of <code>y</code> to <code>b</code>?</p>
<pre>func f() {
 x, y := make([]byte, 9000), make([]byte, 9)
 a := x
 b := y
 // ...
 } </pre>
<p>The answer is; no. <code>x</code> and <code>y</code> have the same type, <code>[]byte</code>, that is, a slice of bytes. As both variables have the same type, their assignment involves copying the same amount of data. Both assignments have the same cost.</p>
<p>All slices are the same size; three machine words (three <code>uintptrs</code>). The first word in the slice is a pointer to the slice’s backing array, the storage for the slice, the second word is the slice’s length, and the third is the capacity. Assigning one slice variable to another copies just three machine words. </p>
<p><em>Further reading: </em><a href="https://blog.golang.org/go-slices-usage-and-internals">Go slices: usage and internals</a> (blog.golang.org)</p>
	</div>
	</article>
	</div></div>||||I|||| Dave Cheney

 The acme of foolishness

 Menu Skip to content
 * High Performance Go
 * Practical Go
 * Internets of Interest
 * About
 + RSS

 A few bytes here, a few there, pretty soon you’re talking real memory

 Today’s post comes from a recent Go pop quiz. Consider this benchmark fragment.1

 func BenchmarkSortStrings(b *testing.B) {
 s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"}
 b.ReportAllocs()
 for i := 0; i < b.N; i++ {
 sort.Strings(s)
 }
 }

 A convenience wrapper around sort.Sort(sort.StringSlice(s)), sort.Strings sorts the input in place, so it isn’t expected to allocate (or at least that’s what 43% of the tweeps who responded thought). However it turns out that, at least in recent versions of Go, each iteration of the benchmark causes one heap allocation. Why does this happen?

 Interfaces, as all Go programmers should know, are implemented as a two word structure. Each interface value contains a field which holds the type of the interface’s contents, and a pointer to the interface’s contents.2

 In pseudo Go code, an interface might look something like this:

 type interface struct {
 // the ordinal number for the type of the value
 // assigned to the interface 
 type uintptr
 
 // (usually) a pointer to the value assigned to
 // the interface
 data uintptr
 }

 interface.data can hold one machine word–8 bytes in most cases–but a []string is 24 bytes; one word for the pointer to the slice’s underlying array; one word for the length; and one for the remaining capacity of the underlying array, so how does Go fit 24 bytes into 8? With the oldest trick in the book, indirection. s, a []string is 24 bytes, but *[]string–a pointer to a slice of strings–is only 8.

 Escaping to the heap

 To make the example a little more explicit, here’s the benchmark rewritten without the sort.Strings helper function:

 func BenchmarkSortStrings(b *testing.B) {
 s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"}
 b.ReportAllocs()
 for i := 0; i < b.N; i++ {
 var ss sort.StringSlice = s
 var si sort.Interface = ss // allocation
 sort.Sort(si)
 }
 }
 

 To make the interface magic work, the compiler rewrites the assignment as var si sort.Interface = &ss–the address of ss is assigned to the interface value.3 We now have a situation where the interface value holds a pointer to ss, but where does it point? Where in memory does ss live?

 It appears that ss is moved to the heap, causing the allocation that the benchmark reports.

 Total: 296.01MB 296.01MB (flat, cum) 99.66%
 8 . . func BenchmarkSortStrings(b *testing.B) { 
 9 . . 	s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"} 
 10 . . 	b.ReportAllocs() 
 11 . . 	for i := 0; i < b.N; i++ { 
 12 . . 		var ss sort.StringSlice = s 
 13 296.01MB 296.01MB 		var si sort.Interface = ss // allocation 
 14 . . 		sort.Sort(si) 
 15 . . 	} 
 16 . . } 

 The allocation occurs because the compiler currently cannot convince itself that ss outlives si. The general attitude amongst Go compiler hackers seems to be that this could be improved, but that’s a discussion for another time. As it stands, ss is allocated on the heap. Thus the question becomes, how many bytes are allocated per iteration? Why don’t we ask the testing package.

 % go test -bench=. sort_test.go
 goos: darwin
 goarch: amd64
 cpu: Intel(R) Core(TM) i7-5650U CPU @ 2.20GHz
 BenchmarkSortStrings-4 12591951 91.36 ns/op 24 B/op 1 allocs/op
 PASS
 ok command-line-arguments 1.260s

 Using Go 1.16beta1, on amd64, 24 bytes are allocated per operation.4 However, the previous Go version, on the same platform, consumes 32 bytes per operation

 % go1.15 test -bench=. sort_test.go
 goos: darwin
 goarch: amd64
 BenchmarkSortStrings-4 11453016 96.4 ns/op 32 B/op 1 allocs/op
 PASS
 ok command-line-arguments 1.225s

 This brings us, circuitously, to the subject of this post, a nifty quality of life improvement coming in Go 1.16. But before I can talk about it, I need to discuss size classes.

 Size classes

 To explain what a size class is, consider how a theoretical Go runtime could allocate 24 bytes on its heap. A simple way to do this would be keep track of all the memory allocated so far using a pointer to the last allocated byte on the heap. To allocate 24 bytes, the heap pointer is incremented by 24, and the previous value returned to the caller. As long as the code that asked for 24 bytes never writes beyond that mark this mechanism has no overhead. Sadly, in real life, memory allocators don’t just allocate memory, sometimes they have to free it.

 Eventually the Go runtime will have to free those 24 bytes, but from the runtime’s point of view, the only thing it knows is the starting address it gave to the caller. It does not know how many bytes after that address were allocated. To permit deallocation, our hypothetical Go runtime allocator would have to record, for each allocation on the heap, its length. Where are the allocation for those lengths allocated? On the heap of course.

 In our scenario, when the runtime wants to allocate memory, it could request a little more than it was asked for and use it to store the amount requested. For our slice example, when we asked for 24 bytes, we’d actually consume 24 bytes plus some overhead to store the number 24. How large is this overhead? It turns out the minimum amount that is practical is one word.5

 To record a 24 byte allocation the overhead would 8 bytes. 25% is not great, but not terrible, and as the size of the allocation goes up, the overhead would become insignificant. However, what would happen if we wanted to store just one byte on the heap? The overhead would be eight times the amount of data we asked for! Is there are more efficient way to allocate small amounts on the heap?

 Rather than storing the length alongside each allocation, what if all the thing of the same size were stored together? If all the 24 byte things are stored together, then the runtime would automatically know how large they are. All the runtime needs is a single bit to indicate if a 24 byte area is in use or not. In Go these areas are known as size classes, because all the things of the same size are stored together (think school class–all the students are the same grade–not a C++ class). When the runtime needs to allocate a small amount, it does so using the smallest size class that can accomodate the allocation.

 Unlimited size classes for all

 Now we know how size classes work, the obvious question is, where are they stored? Not surprisingly, the memory for a size class comes from the heap. To minimise overhead, the runtime allocates a larger amount from the heap (usually a multiple of the system page size) then dedicates that space for allocations of a single size. But, there’s a problem.

 The pattern of allocating a large area to store things of the same size works well6 if there’s a fixed, preferably small, number of allocation sizes, but in a general purpose language programs can ask the runtime for an allocation of any size.7

 For example, imagine asking the runtime for 9 bytes. 9 bytes is an uncommon size, so it’s likely that a new size class for things of size 9 would be required. As 9 byte things are uncommon, it’s likely the remainder of the allocation, 4kb or more, would be wasted. Because of this the set of possible size classes is fixed. If a size class of the exact amount is not available, the allocation is rounded up to the next size class. In our example 9 bytes might be allocated in the 12 byte size class. The overhead of 3 unused bytes is better than an entire size class allocation which goes mostly unused.

 All together now

 This is the final piece of the puzzle. Go 1.15 did not have a 24 byte size class, so the heap allocation of ss was allocated in the 32 byte size class. Thanks to the work of Martin Möhrmann Go 1.16 has a 24 byte size class, which is a perfect fit for slice values assigned to interfaces.

 This entry was posted in Go, Programming on January 5, 2021 by Dave Cheney.

 The story of the one line fix

 Picture yourself, an engineer working at the hottest distributed microservices de jour, assigned to fix a bug. You jump into an unfamiliar codebase and quickly locate the line where the problem occurred. The fix is simple, just return early or substitute a default value in the case that one cannot be determined from your input. Boom, problem solved. Code compiles, fix goes out to production, and the card’s in the done pile in time for Tuesday drinks at the pub.

 Now consider this alternative scenario. You find the problematic line and, before you make the fix, you decide to add a test so you will know that your fix worked, and hopefully someone will not accidentally revert it in the future.

 You’ve figured out that some weird edge case causes that line to be called without being able to determine the right value for y. You can see it clearly, y ends up being zero so the program crashes. To write a test, you need to get the conditions that caused y to be zero to occur on command. There are only two parameters passed into this function, this shouldn’t be hard. Oh, but this is a method, not a function, so you need to construct an instance of the object in the right state to trigger the bug. Hmm, this language uses constructors to make sure people can’t just monkey up the bits of the object they need for a test, you’ll have to find, mock, stub or build instances of all the objects dependencies (and their dependencies, and so on) then run the object through the precise series of operations to put the it in the state that causes y to be empty. Then you can write the test.

 At this point Tuesday night drinks are fading before you eyes. Without being able to write a test for your fix, how will you convince a reviewer that it actually fixed what it fixed? All you can prove was your “simple” change didn’t break any behaviour that was covered by existing tests.

 Fast forward to Friday and your one line change has finally been merged, along with a few new test cases to make sure it doesn’t happen again. Next week you’re going to sit down and try to figure out a bunch of weird crashes you caused while trying to stub out calls to the database. Eventually you gave up and copied some code from another test that uses a copy of production data. It’s much slower than it needs to be, but it worked, and was the only way to put a reasonable estimate on a job that had already consumed your entire week.

 The moral of the story; was the test the reason it took nearly a week to make a one line fix?

 This entry was posted in Programming, Small ideas on December 15, 2020 by Dave Cheney.

 How to dump the GOSSAFUNC graph for a method

 The Go compiler’s SSA backend contains a facility to produce HTML debugging output of the compilation phases. This post covers how to print the SSA output for function and methods.

 Let’s start with a sample program which contains a function, a value method, and a pointer method:

 package main
 
 import (
 "fmt"
 )
 
 type Numbers struct {
 vals []int
 }
 
 func (n *Numbers) Add(v int) {
 n.vals = append(n.vals, v)
 }
 
 func (n Numbers) Average() float64 {
 sum := 0.0
 for _, num := range n.vals {
 sum += float64(num)
 }
 return sum / float64(len(n.vals))
 }
 
 
 func main() {
 var numbers Numbers
 numbers.Add(200)
 numbers.Add(43)
 numbers.Add(-6)
 fmt.Println(numbers.Average())
 }

 Control of the SSA debugging output is via the GOSSAFUNC environment variable. This variable takes the name of the function to dump. This is not the functions fully qualified name. For func main above the name of the function is main not main.main.

 % env GOSSAFUNC=main go build
 runtime
 dumped SSA to ../../go/src/runtime/ssa.html
 t
 dumped SSA to ./ssa.html

 In this example GOSSAFUNC=main matched both main.main and a function called runtime.main.1 This is a little unfortunate, but in practice probably not a big deal as, if you’re performance tuning your code, it won’t be in a giant spaghetti blob in func main.

 What is more likely is your code will be in a method, so you’ve probably landed on this post looking for the correct incantation to dump the SSA output for a method.

 To print the SSA debug for the pointer method func (n *Numbers) Add, the equivalent function name is(*Numbers).Add:2

 % env "GOSSAFUNC=(*Numbers).Add" go build
 t
 dumped SSA to ./ssa.html

 To print the SSA debug for a value method func (n Numbers) Average, the equivalent function name is (*Numbers).Average even though this is a value method:

 % env "GOSSAFUNC=(*Numbers).Average" go build
 t
 dumped SSA to ./ssa.html
 This entry was posted in Go, Programming and tagged compiler, gossafunc, ssa on June 19, 2020 by Dave Cheney.

 Diamond interface composition in Go 1.14

 Per the overlapping interfaces proposal, Go 1.14 now permits embedding of interfaces with overlapping method sets. This is a brief post explain what this change means:

 Let’s start with the definition of the three key interfaces from the io package; io.Reader, io.Writer, and io.Closer:

 package io
 
 type Reader interface {
 Read([]byte) (int, error)
 }
 
 type Writer interface {
 Write([]byte) (int, error)
 }
 
 type Closer interface {
 Close() error
 } 

 Just as embedding a type inside a struct allows the embedded type’s fields and methods to be accessed as if it were declared on the embedding type1, the process is true for interfaces. Thus there is no difference between explicitly declaring

 type ReadCloser interface {
 Read([]byte) (int, error)
 Close() error
 }

 and using embedding to compose the interface

 type ReadCloser interface {
 Reader
 Closer
 }

 You can even mix and match

 type WriteCloser interface {
 Write([]byte) (int, error)
 Closer
 }

 However, prior to Go 1.14, if you continued to compose interface declarations in this manner you would likely find that something like this,

 type ReadWriteCloser interface {
 ReadCloser
 WriterCloser
 }

 would fail to compile

 % go build interfaces.go
 command-line-arguments
 ./interfaces.go:27:2: duplicate method Close

 Fortunately, with Go 1.14 this is no longer a limitation, thus solving problems that typically occur with diamond-shaped embedding graphs.

 However, there is a catch that I ran into attempting to demonstrate this feature to the local user group–this feature is only enabled when the Go compiler uses the 1.14 (or later) spec.

 As near as I can make out the rules for which version of the Go spec is used during compilation appear to be:

 1. If your source code is stored inside GOPATH (or you have disabled modules with GO111MODULE=off) then the version of the Go spec used to compile with matches the version of the compiler you are using. Said another way, if you have Go 1.13 installed, your Go version is 1.13. If you have Go 1.14 installed, your version is 1.14. No surprises here.
 2. If your source code is stored outside GOPATH (or you have forced modules on with GO111MODULE=on) then the go tool will take the Go version from the go.mod file.
 3. If there is no Go version listed in go.mod then the version of the spec will be the version of Go installed. This is identical to point 1.
 4. If you are in module mode, either by being outside GOPATH or with GO111MODULE=on, but there is no go.mod file in the current, or any parent, directory then the version of the Go spec used to compile your code defaults to Go 1.13.

 The last point caught me out.

 This entry was posted in Go, Programming and tagged compiler, interfaces on May 24, 2020 by Dave Cheney.

 Fatih’s question

 A few days ago Fatih posted this question on twitter.

 I’m going to attempt to give my answer, however to do that I need to apply some simplifications as my previous attempts to answer it involved a lot of phrases like a pointer to a pointer, and other unhelpful waffling. Hopefully my simplified answer can be useful in building a mental framework to answer Fatih’s original question.

 Restating the question

 Fatih’s original tweet showed four different variations of json.Unmarshal. I’m going to focus on the last two, which I’ll rewrite a little:

 package main
 
 import (
 "encoding/json"
 "fmt"
 )
 
 type Result struct {
 Foo string `json:"foo"`
 }
 
 func main() {
 content := []byte(`{"foo": "bar"}`)
 var result1, result2 *Result
 
 err := json.Unmarshal(content, &result1)
 fmt.Println(result1, err) // &{bar} <nil>
 
 err = json.Unmarshal(content, result2)
 fmt.Println(result2, err) // <nil> json: Unmarshal(nil *main.Result)
 }

 Restated in words, result1 and result2 are the same type; *Result. Decoding into result1 works as expected, whereas decoding into result2 causes the json package to complain that the value passed to Unmarshal is nil. However, both values were declared without an initialiser so both would have taken on the type’s zero value, nil.

 Eagle eyed readers will have spotted that the reason for the difference is the first invocation is passed &result1, while the second is passed result2, but this explanation is unsatisfactory because the documentation for json.Unmarshal states:

 Unmarshal parses the JSON-encoded data and stores the result in the value pointed to by v. If v is nil or not a pointer, Unmarshal returns an InvalidUnmarshalError.

 Which is confusing because result1 and result2 are pointers. Furthermore, without initialisation, both are nil. Now, the documentation is correct (as you’d expect from a package that has been hammered on for a decade), but explaining why takes a little more investigation.

 Functions receive a copy of their arguments

 Every assignment in Go is a copy, this includes function arguments and return values.

 package main
 
 import (
 "fmt"
 )
 
 func increment(v int) {
 v++
 }
 
 func main() {
 v := 1
 increment(v)
 fmt.Println(v) // 1
 }

 In this example, increment is operating on a copy of main‘s v. This is because the v declared in main and increment‘s v parameter have different addresses in memory. Thus changes to increment‘s v cannot affect the contents of main‘s v.

 package main
 
 import (
 "fmt"
 )
 
 func increment(v *int) {
 *v++
 }
 
 func main() {
 v := 1
 increment(&v)
 fmt.Println(v) // 2
 }

 If we wanted to write increment in a way that it could affect the contents of its caller we would need to pass a reference, a pointer, to main.v.1 This example demonstrates why json.Unmarshal needs a pointer to the value to decode JSON into.

 Pointers to pointers

 Returning to the original question, both result1 and result2 are declared as *Result, that is, pointers to a Result value. We established that you have to pass the address of caller’s value to json.Unmarshal otherwise it won’t be able to alter the contents of the caller’s value. Why then must we pass the address of result1, a **Result, a pointer to a pointer to a Result, for the operation to succeed.

 To explain this another detour is required. Consider this code:

 package main
 
 import (
 "encoding/json"
 "fmt"
 )
 
 type Result struct {
 Foo *string `json:"foo"`
 }
 
 func main() {
 content := []byte(`{"foo": "bar"}`)
 var result1 *Result
 
 err := json.Unmarshal(content, &result1)
 fmt.Printf("%#v %v", result1, err) // &main.Result{Foo:(*string)(0xc0000102f0)} <nil>
 }

 In this example Result contains a pointer typed field, Foo *string. During JSON decoding Unmarshal allocated a new string value, stored the value bar in it, then placed the address of the string in Result.Foo. This behaviour is quite handy as it frees the caller from having to initialise Result.Foo and makes it easier to detect when a field was not initialised because the JSON did not contain a value. Beyond the convenience this offers for simple examples it would be prohibitively difficult for the caller to properly initialise all the reference type fields in a structure before decoding unknown JSON without first inspecting the incoming JSON which itself may be problematic if the input is coming from an io.Reader without the ability to rewind the input.

 To unmarshal JSON into a pointer, Unmarshal first handles the case of the JSON being the JSON literal null. In that case, Unmarshal sets the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the value pointed at by the pointer. If the pointer is nil, Unmarshal allocates a new value for it to point to.

 json.Unmarshal‘s handling of pointer fields is clearly documented, and works as you would expect, allocating a new value whenever there is a need to decode into a pointer shaped field. It is this behaviour that gives us a hint to what is happening in the original example.

 We’ve seen that when json.Unmarshal encounters a field which points to nil it will allocate a new value of the correct type and assign its address the field before proceeding. Not only is does behaviour is applied recursively–for example in the case of a complex structure which contains pointers to other structures–but it also applies to the value passed to Unmarshal.

 package main
 
 import (
 "encoding/json"
 "fmt"
 )
 
 func main() {
 content := []byte(`1`)
 var result *int
 
 err := json.Unmarshal(content, &result)
 fmt.Println(*result, err) // 1 <nil>
 }

 In this example result is not a struct, but a simple *int which, lacking an initialiser, defaults to nil. After JSON decoding, result now points to an int with the value 1.

 Putting the pieces together

 Now I think I’m ready to take a shot at answering Fatih’s question.

 json.Unmarshal requires the address of the variable you want to decode into, otherwise it would decode into a temporary copy which would be discard on return. Normally this is done by declaring a value, then passing its address, or explicitly initialising the the value

 var result1 Result
 err := json.Unmarshal(content, &result1) // this is fine
 
 var result2 = new(Result)
 err = json.Unmarshal(content, result2) // and this
 
 var result3 = &Result{}
 err = json.Unmarshal(content, result3) // this is also fine

 In all three cases the address that the *Result points too is not nil, it points to initialised memory that json.Unmarshal decodes into.

 Now consider what happens when json.Unmarshal encounters this

 var result4 *Result
 err = json.Unmarshal(content, result4) // err json: Unmarshal(nil *main.Result)

 result2, result3, and the expression &result1 point to a Result. However result4, even though it has the same type as the previous three, does not point to initialised memory, it points to nil. Thus, according to the examples we saw previously, before json.Unmarshal can decode into it, the memory result4 points too must be initialised.

 However, because each function receives a copy of its arguments, the caller’s result4 variable and the copy inside json.Unmarshal are unique. json.Unmarshal can allocate a new Result value and decode into it, but it cannot alter result4 to point to this new value because it was not provided with a reference to result4, only a copy of its contents.

 This entry was posted in Go, Programming and tagged json on May 16, 2020 by Dave Cheney.

 Ensmallening Go binaries by prohibiting comparisons

 Conventional wisdom dictates that the larger the number of types declared in a Go program, the larger the resulting binary. Intuitively this makes sense, after all, what’s the point in defining a bunch of types if you’re not going to write code that operates on them. However, part of the job of a linker is to detect functions which are not referenced by a program–say they are part of a library of which only a subset of functionality is used–and remove them from the final output. Yet, the adage mo’ types, mo’ binary holds true for the majority of Go programs.

 In this post I’ll dig into what equality, in the context of a Go program, means and why changes like this have a measurable impact on the size of a Go program.

 Defining equality between two values

 The Go spec defines the concepts of assignability and equality. Assignabiity is the act of assigning a value to an identifier. Not everything which is declared can be assigned, for example constants and functions. Equality is the act of comparing two identifies by asking are their contents the same?

 Being a strongly typed language, the notion of sameness is fundamentally rooted in the identifier’s type. Two things can only be the same if they are of the same type. Beyond that, the type of the values defines how they are compared.

 For example, integers are compared arithmetically. For pointer types, equality is determining if the addresses they point too are the same. Reference types like maps and channels, like pointers, are considered to be the same if they have the same address.

 These are all examples of bitwise equality, that is, if the bit patterns of the memory that value occupies are the same, those values are equal. This is known as memcmp, short for memory comparison, as equality is defined by comparing the contents of two areas of memory.

 Hold on to this idea, I’ll come back to in a second.

 Struct equality

 Beyond scalar types like integers, floats, and pointers is the realm of compound types; structs. All structs are laid out in memory in program order, thus this declaration:

 type S struct {
 a, b, c, d int64
 }

 will consume 32 bytes of memory; 8 bytes for a, then 8 bytes for b, and so on. The spec says that struct values are comparable if all their fields are comparable. Thus two structs are equal iff each of their fields are equal.

 a := S{1, 2, 3, 4}
 b := S{1, 2, 3, 4}
 fmt.Println(a == b) // prints true

 Under the hood the compiler uses memcmp to compare the 32 bytes of a and b.

 Padding and alignment

 However the simplistic bitwise comparison strategy will fail in situations like this:

 type S struct {
 a byte
 b uint64
 c int16
 d uint32
 }
 
 func main()
 a := S{1, 2, 3, 4}
 b := S{1, 2, 3, 4}
 fmt.Println(a == b) // prints true
 }

 The code compiles, the comparison is still true, but under the hood the compiler cannot rely on comparing the bit patterns of a and b because the structure contains padding.

 Go requires each field in a struct to be naturally aligned. 2 byte values must start on an even address, four byte values on an address divisible by 4, and so on1. The compiler inserts padding to ensure the fields are aligned to according to their type and the underlying platform. In effect, after padding, this is what the compiler sees2:

 type S struct {
 a byte
 _ [7]byte // padding
 b uint64
 c int16
 _ [2]int16 // padding
 d uint32
 }

 Padding exists to ensure the correct field alignments, and while it does take up space in memory, the contents of those padding bytes are unknown. You might assume that, being Go, the padding bytes are always zero, but it turns out that’s not the case–the contents of padding bytes are simply not defined. Because they’re not defined to always be a certain value, doing a bitwise comparison may return false because the nine bytes of padding spread throughout the 24 bytes of S are may not be the same.

 The Go compiler solves this problem by generating what is known as an equality function. In this case S‘s equality function knows how to compare two values of type S by comparing only the fields in the function while skipping over the padding.

 Type algorithms

 Phew, that was a lot of setup to illustrate why, for each type defined in a Go program, the compiler may generate several supporting functions, known inside the compiler as the type’s algorithms. In addition to the equality function the compiler will generate a hash function if the type is used as a map key. Like the equality function, the hash function must consider factors like padding when computing its result to ensure it remains stable.

 It turns out that it can be hard, and sometimes non obvious, to intuit when the compiler will generate these functions–it’s more than you’d expect–and it can be hard for the linker to eliminate the ones that are not needed as reflection often causes the linker to be more conservative when trimming types.

 Reducing binary size by prohibiting comparisons

 Now we’re at a point to explain Brad’s change. By adding an incomparable field 3 to the type, the resulting struct is by extension incomparable, thus forcing the compiler to elide the generation of eq and hash algorithms, short circuiting the linkers elimination of those types and, in practice, reducing the size of the final binary. As an example of this technique, this program:

 package main
 
 import "fmt"
 
 func main() {
 type t struct {
 // _ [0][]byte uncomment to prevent comparison
 a byte
 b uint16
 c int32
 d uint64
 }
 var a t
 fmt.Println(a)
 }

 when compiled with Go 1.14.2 (darwin/amd64), decreased from 2174088 to 2174056, a saving of 32 bytes. In isolation this 32 byte saving may seem like small beer, but consider that equality and hash functions can be generated for every type in the transitive closure of your program and all its dependencies, and the size of these functions varies depending on the size of the type and its complexity, prohibiting them can have a sizeable impact on the final binary over and above the old saw of -ldflags="-s -w".

 The bottom line, if you don’t wish to make your types comparable, a hack like this enforces it at the source level while contributing to a small reduction in the size of your binary.

 Addendum: thanks to Brad’s prodding, Go 1.15 already has a bunch of improvements by Cherry Zhang and Keith Randall that fix the most egregious of the failures to eliminate unnecessary equality and hash functions (although I suspect it was also to avoid the proliferation of this class of CLs).

 This entry was posted in Go, Programming on May 9, 2020 by Dave Cheney.

 Mid-stack inlining in Go

 In the previous post I discussed how leaf inlining allows the Go compiler to reduce the overhead of function calls and extend optimisation opportunities across function boundaries. In this post I’ll discuss the limits of inlining and leaf vs mid-stack inlining.

 The limits of inlining

 Inlining a function into its caller removes the call’s overhead and increases the opportunity for the compiler to apply additional optimisations so the question should be asked, if some inlining is good, would more be better, why not inline as much as possible?

 Inlining trades possibly larger program sizes for potentially faster execution time. The main reason to limit inlining is creating many inlined copies of a function can increase compile time and result in larger binaries for marginal gain. Even taking into account the opportunities for further optimisation, aggressive inlining tends to increase the size of, and the time too compile, the resulting binary.

 Inlining works best for small functions that do relatively little work compared to the overhead of calling them. As the size of a function grows, the time saved avoiding the call’s overhead diminishes relative to the work done inside the function. Larger functions tend to be more complex, thus the benefits of optimising their inlined forms vs in situ are reduced.

 Inlining budget

 During compilation each function’s inlineabilty is calculated using what is known as the inlining budget1. The cost calculation can be tricky to internalise but is broadly one unit per node in the AST for simple things like unary and binary operations but can be higher for complex operations like make. Consider this example:

 package main
 
 func small() string {
 s := "hello, " + "world!"
 return s
 }
 
 func large() string {
 s := "a"
 s += "b"
 s += "c"
 s += "d"
 s += "e"
 s += "f"
 s += "g"
 s += "h"
 s += "i"
 s += "j"
 s += "k"
 s += "l"
 s += "m"
 s += "n"
 s += "o"
 s += "p"
 s += "q"
 s += "r"
 s += "s"
 s += "t"
 s += "u"
 s += "v"
 s += "w"
 s += "x"
 s += "y"
 s += "z"
 return s
 }
 
 func main() {
 small()
 large()
 }

 Compiling this function with -gcflags=-m=2 allows us to see the cost the compiler assigns to each function.

 % go build -gcflags=-m=2 inl.go 
 # command-line-arguments
 ./inl.go:3:6: can inline small with cost 7 as: func() string { s := "hello, world!"; return s }
 ./inl.go:8:6: cannot inline large: function too complex: cost 82 exceeds budget 80
 ./inl.go:38:6: can inline main with cost 68 as: func() { small(); large() }
 ./inl.go:39:7: inlining call to small func() string { s := "hello, world!"; return s }

 The compiler determined that func small() can be inlined due to its cost of 7. func large() was determined to be too expensive. func main()has been marked as eligible and assigned a cost of 68; 7 from the body of small, 57 from the function call to small and the remainder in its own overhead.

 The inlining budget can be controlled to some degree with the -gcflag=-l flag. Currently the values that apply are:

 * -gcflags=-l=0 is the default level of inlining.
 * -gcflags=-l (or -gcflags=-l=1) disables inlining.
 * -gcflags=-l=2 and -gcflags=-l=3 are currently unused and have no effect over -gcflags=-l=0
 * -gcflags=-l=4 reduces the cost for inlining non-leaf functions and calls through interfaces.2

 Hairy optimisations

 Some functions with a relatively low inlining cost may be ineligible because of their complexity. This is known as the function’s hairiness as the semantics of some operations are hard to reason about once inlined, for example recover, break. Others, like select and go, involve co-ordination with the runtime so the extra effort of inlining doesn’t pay for itself.

 The list of hairy statements also includes things like for and range which don’t have an inherently large cost, but simply haven’t been optimised yet.

 Mid stack inlining

 Historically the Go compiler only performed leaf inlining–only functions which did not call other functions were eligible. In the context of the hairiness discussion previously, a function call would disqualify the function from being inlined.

 Enter mid stack inlining which, as its name implies, allows functions in the middle of a call stack to be inlined without requiring everything below them to be eligible. Mid stack inlining was introduced by David Lazar in Go 1.9 and improved in subsequent releases. This presentation goes into some of the difficulties with retaining the behaviour of stack traces and runtime.Callers in code paths that had been heavily inlined.

 We see an example of mid-stack inlining in the previous example. After inlining, func main() contains the body of func small() and a call to func large(), thus it is considered a non-leaf function. Historically this would have prevented it from being further inlined even though its combined cost was less than the inlining budget.

 The primary use case for mid stack inlining is to reduce the overhead of a path through the call stack. Consider this example:

 package main
 
 import (
 "fmt"
 "strconv"
 )
 
 type Rectangle struct {}
 
 //go:noinline
 func (r *Rectangle) Height() int {
 h, _ := strconv.ParseInt("7", 10, 0)
 return int(h)
 }
 
 func (r *Rectangle) Width() int {
 return 6
 }
 
 func (r *Rectangle) Area() int { return r.Height() * r.Width() }
 
 func main() {
 var r Rectangle
 fmt.Println(r.Area())
 }

 In this example r.Area() is a simple function which calls two others. r.Width() can be inlined while r.Height(), simulated here with the //go:noinline annotation, cannot. 3

 % go build -gcflags='-m=2' square.go  
 # command-line-arguments
 ./square.go:12:6: cannot inline (*Rectangle).Height: marked go:noinline  
 ./square.go:17:6: can inline (*Rectangle).Width with cost 2 as: method(*Rectangle) func() int { return 6 }
 ./square.go:21:6: can inline (*Rectangle).Area with cost 67 as: method(*Rectangle) func() int { return r.Height() * r.Width() } ./square.go:21:61: inlining call to (*Rectangle).Width method(*Rectangle) func() int { return 6 } 
 ./square.go:23:6: cannot inline main: function too complex: cost 150 exceeds budget 80 
 ./square.go:25:20: inlining call to (*Rectangle).Area method(*Rectangle) func() int { return r.Height() * r.Width() }
 ./square.go:25:20: inlining call to (*Rectangle).Width method(*Rectangle) func() int { return 6 }

 As the multiplication performed by r.Area() is cheap compared to the overhead of calling it, inlining r.Area()‘s single expression is a net win even if its downstream caller to r.Height() remains ineligible.

 Fast path inlining

 The most startling example of the power of mid-stack inlining comes from 2019 when Carlo Alberto Ferraris improved the performance of sync.Mutex.Lock() by allowing the fast path of the lock–the uncontended case–to be inlined into its caller. Prior to this change sync.Mutex.Lock() was a large function containing many hairy conditions which made it ineligible to be inlined. Even in the case where the lock was available, the caller had to pay the overhead of calling sync.Mutex.Lock().

 Carlo’s change split sync.Mutex.Lock() into two functions (a process he dubbed outlining). The outer sync.Mutex.Lock() method now calls sync/atomic.CompareAndSwapInt32() and returns to the caller immediately if the CAS succeeds. If not, the function falls through to sync.Mutex.lockSlow() which handles the slow path required to register interest on the lock and park the goroutine.4

 % go build -gcflags='-m=2 -l=0' sync 2>&1 | grep '(*Mutex).Lock'
 ../go/src/sync/mutex.go:72:6: can inline (*Mutex).Lock with cost 69 as: method(*Mutex) func() { if "sync/atomic".CompareAndSwapInt32(&m.state, 0, mutexLocked) { if race.Enabled {  }; return  }; m.lockSlow() }

 By splitting the function into an easily inlineable outer function, falling through to a complex inner function to handle the slow path Carlo’s combined mid stack inlining and the compiler’s support for intrinsic operations to reduce the cost of an uncontended lock by 14%. Then he repeated the trick for an additional 9% saving in sync.RWMutex.Unlock().

 This entry was posted in Go, Programming and tagged inlining, optimisation, performance on May 2, 2020 by Dave Cheney.

 Inlining optimisations in Go

 This is a post about how the Go compiler implements inlining and how this optimisation affects your Go code.

 n.b. This article focuses on gc, the de facto Go compiler from golang.org. The concepts discussed apply broadly to other Go compilers like gccgo and tinygo but may differ in implementation and efficacy.

 What is inlining?

 Inlining is the act of combining smaller functions into their respective callers. In the early days of computing this optimisation was typically performed by hand. Nowadays inlining is one of a class of fundamental optimisations performed automatically during the compilation process.

 Why is inlining important?

 Inlining is important for two reasons. The first is it removes the overhead of the function call itself. The second is it permits the compiler to more effectively apply other optimisation strategies.

 Function call overhead

 Calling a function1 in any language carries a cost. There are the overheads of marshalling parameters into registers or onto the stack (depending on the ABI) and reversing the process on return. Invoking a function call involves jumping the program counter from one point in the instruction stream to another which can cause a pipeline stall. Once inside the function there is usually some preamble required to prepare a new stack frame for the function to execute and a similar epilogue needed to retire the frame before returning to the caller.

 In Go a function call carries additional costs to support dynamic stack growth. On entry the amount of stack space available to the goroutine is compared to the amount required for the function. If insufficient stack space is available, the preamble jumps into the runtime logic that grows the stack by copying it to a new, larger, location. Once this is done the runtime jumps back to the start of the original function, the stack check is performed again, which now passes, and the call continues. In this way goroutines can start with a small stack allocation which grows only when needed.2

 This check is cheap–only a few instructions–and because goroutine stacks grows geometrically the check rarely fails. Thus, the branch prediction unit inside a modern processor can hide the cost of the stack check by assuming it will always be successful. In the case where the processor mis-predicts the stack check and has to discard the work done while it was executing speculatively, the cost of the pipeline stall is relatively small compared to the cost of the work needed for the runtime to grow a goroutines stack.

 While the overhead of the generic and Go specific components of each function call are well optimised by modern processors using speculative execution techniques, those overheads cannot be entirely eliminated, thus each function call carries with it a performance cost over and above the time it takes to perform useful work. As a function call’s overhead is fixed, smaller functions pay a larger cost relative to larger ones because they tend to do less useful work per invocation.

 The solution to eliminating these overheads must therefore be to eliminate the function call itself, which the Go compiler does, under certain conditions, by replacing the call to a function with the contents of the function. This is known as inlining because it brings the body of the function in line with its caller.

 Improved optimisation opportunities

 Dr. Cliff Click describes inlining as the optimisation performed by modern compilers as it forms the basis for optimisations like constant propagation and dead code elimination. In effect, inlining allows the compiler to see further, allowing it to observe, in the context that a particular function is being called, logic that can be further simplified or eliminated entirely. As inlining can be applied recursively optimisation decisions can be made not only in the context of each individual function, but also applied to the chain of functions in a call path.

 Inlining in action

 The effects of inlining can be demonstrated with this small example

 package main
 
 import "testing"
 
 //go:noinline
 func max(a, b int) int {
 if a > b {
 return a
 }
 return b
 }
 
 var Result int
 
 func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i < b.N; i++ {
 r = max(-1, i)
 }
 Result = r
 }

 Running this benchmark gives the following result:3

 % go test -bench=. 
 BenchmarkMax-4   530687617         2.24 ns/op

 The cost of max(-1, i) is around 2.24 nanoseconds on my 2015 MacBook Air. Now let’s remove the //go:noinline pragma and see the result:

 % go test -bench=. 
 BenchmarkMax-4   1000000000         0.514 ns/op

 From 2.24 ns to 0.51 ns, or according to benchstat, a 78% improvement.

 % benchstat {old,new}.txt
 name   old time/op  new time/op  delta
 Max-4  2.21ns ± 1%  0.49ns ± 6%  -77.96%  (p=0.000 n=18+19)

 Where did these improvements come from?

 First, the removal of the function call and associated preamble4 was a major contributor. Pulling the contents of max into its caller reduced the number of instructions executed by the processor and eliminated several branches.

 Now the contents of max are visible to the compiler as it optimises BenchmarkMax it can make some additional improvements. Consider that once max is inlined, this is what the body of BenchmarkMax looks like to the compiler:

 func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i < b.N; i++ {
 if -1 > i {
 r = -1
 } else {
 r = i
 }
 }
 Result = r
 }

 Running the benchmark again we see our manually inlined version performs as well as the version inlined by the compiler

 % benchstat {old,new}.txt
 name   old time/op  new time/op  delta
 Max-4  2.21ns ± 1%  0.48ns ± 3%  -78.14%  (p=0.000 n=18+18)

 Now the compiler has access to the result of inlining max into BenchmarkMax it can apply optimisation passes which were not possible before. For example, the compiler has noted that i is initialised to 0 and only incremented so any comparison with i can assume i will never be negative. Thus, the condition -1 > i will never be true.5

 Having proved that -1 > i will never be true, the compiler can simplify the code to

 func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i < b.N; i++ {
 if false {
 r = -1
 } else {
 r = i
 }
 }
 Result = r
 }

 and because the branch is now a constant, the compiler can eliminate the unreachable path leaving it with

 func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i < b.N; i++ {
 r = i
 }
 Result = r
 }

 Thus, through inlining and the optimisations it unlocks, the compiler has reduced the expression r = max(-1, i) to simply r = i.

 The limits of inlining

 In this article I’ve discussed, so called, leaf inlining; the act of inlining a function at the bottom of a call stack into its direct caller. Inlining is a recursive process, once a function has been inlined into its caller, the compiler may inline the resulting code into its caller, as so on. For example, this code

 func BenchmarkMaxMaxMax(b *testing.B) {
 var r int
 for i := 0; i < b.N; i++ {
 r = max(max(-1, i), max(0, i))
 }
 Result = r
 }

 runs as fast as the previous example as the compiler is able to repeatedly apply the optimisations outlined above to reduce the code to the same r = i expression.

 In the next article I’ll discuss an alternative inlining strategy when the Go compiler wishes to inline a function in the middle of a call stack. Finally I’ll discuss the limits that the compiler is prepared to go to to inline code, and which Go constructs are currently beyond its capability.

 This entry was posted in Go, Programming and tagged inlining, optimisation, performance on April 25, 2020 by Dave Cheney.

 go test -v streaming output

 The testing package is one of my favourite packages in the Go standard library, not just because of its low noise approach to unit testing, but, over the lifetime of Go, it has received a steady stream of quality of life improvements driven by real world usage.

 The most recent example of this is, in Go 1.14, go test -v will stream t.Log output as it happens, rather than hoarding it til the end of the test run. Here’s an example;

 package main
 
 import (
 	"fmt"
 	"testing"
 	"time"
 )
 
 func TestLogStreaming(t *testing.T) {
 	for i := 0; i < 5; i++ {
 		time.Sleep(300 * time.Millisecond)
 		fmt.Println("fmt.Println:", i)
 		t.Log("t.Log:", i)
 	}
 }

 Note: Calling fmt.Println inside a test is generally considered a no no as it bypasses the testing package’s output buffering irrespective of the -v flag. However, for this example, it‘s necessary to demonstrate the streaming t.Log change.

 % go1.13 test -v tlog_test.go
 === RUN TestLogStreaming
 fmt.Println: 0
 fmt.Println: 1
 fmt.Println: 2
 fmt.Println: 3
 fmt.Println: 4
 --- PASS: TestLogStreaming (1.52s)
 tlog_test.go:13: t.Log: 0
 tlog_test.go:13: t.Log: 1
 tlog_test.go:13: t.Log: 2
 tlog_test.go:13: t.Log: 3
 tlog_test.go:13: t.Log: 4
 PASS
 ok command-line-arguments 1.971s

 Under Go 1.13 and earlier the fmt.Println lines output immediately. t.Log lines are buffered and are printed after the test completes.

 % go1.14 test -v tlog_test.go
 === RUN TestLogStreaming
 fmt.Println: 0
 TestLogStreaming: tlog_test.go:13: t.Log: 0
 fmt.Println: 1
 TestLogStreaming: tlog_test.go:13: t.Log: 1
 fmt.Println: 2
 TestLogStreaming: tlog_test.go:13: t.Log: 2
 fmt.Println: 3
 TestLogStreaming: tlog_test.go:13: t.Log: 3
 fmt.Println: 4
 TestLogStreaming: tlog_test.go:13: t.Log: 4
 --- PASS: TestLogStreaming (1.51s)
 PASS
 ok command-line-arguments 1.809s

 Under Go 1.14 the fmt.Println and t.Log lines are interleaved, rather than waiting for the test to complete, demonstrating that test output is streamed when go test -v is used.

 This is a great quality of life improvement for integration style tests that often retry for long periods when the test is failing. Streaming t.Log output will help Gophers debug those test failures without having to wait until the entire test times out to receive their output.

 This entry was posted in Go, Programming and tagged testing, unit test on March 10, 2020 by Dave Cheney.

 Are large slices more expensive than smaller ones?

 Programmers have a tendency to be superstitious. Particularly, when a programmer hears that copies are expensive, they start to see them everywhere, especially when they learn that, in Go, every assignment is a copy.

 Consider this code; x is three orders of magnitude larger than y, is the assignment of x to a more expensive than the assignment of y to b?

 func f() {
 x, y := make([]byte, 9000), make([]byte, 9)
 a := x
 b := y
 // ...
 } 

 The answer is; no. x and y have the same type, []byte, that is, a slice of bytes. As both variables have the same type, their assignment involves copying the same amount of data. Both assignments have the same cost.

 All slices are the same size; three machine words (three uintptrs). The first word in the slice is a pointer to the slice’s backing array, the storage for the slice, the second word is the slice’s length, and the third is the capacity. Assigning one slice variable to another copies just three machine words.

 Further reading: Go slices: usage and internals (blog.golang.org)

 This entry was posted in Go and tagged performance, slices on March 1, 2020 by Dave Cheney.

 Post navigation

 ← Older posts

 Elsewhere

 * @davecheney
 * LinkedIn
 * Serverfault
 * Stackoverflow

 Categories

 * Economy
 * Go
 * Hardware Hacking
 * History
 * Internets of interest
 * Photography
 * Programming
 * Retrochallenge
 * Small ideas
 * Uncategorized
 * Useless Trivia

 Archives

 * January 2021
 * December 2020
 * June 2020
 * May 2020
 * April 2020
 * March 2020
 * February 2020
 * December 2019
 * November 2019
 * October 2019
 * September 2019
 * August 2019
 * July 2019
 * June 2019
 * May 2019
 * April 2019
 * February 2019
 * January 2019
 * December 2018
 * November 2018
 * October 2018
 * September 2018
 * August 2018
 * July 2018
 * May 2018
 * January 2018
 * December 2017
 * November 2017
 * September 2017
 * August 2017
 * July 2017
 * June 2017
 * April 2017
 * March 2017
 * February 2017
 * January 2017
 * December 2016
 * November 2016
 * October 2016
 * September 2016
 * August 2016
 * June 2016
 * May 2016
 * April 2016
 * March 2016
 * February 2016
 * January 2016
 * December 2015
 * November 2015
 * October 2015
 * September 2015
 * August 2015
 * July 2015
 * June 2015
 * May 2015
 * March 2015
 * February 2015
 * January 2015
 * December 2014
 * November 2014
 * October 2014
 * September 2014
 * August 2014
 * July 2014
 * June 2014
 * May 2014
 * April 2014
 * March 2014
 * February 2014
 * January 2014
 * December 2013
 * November 2013
 * October 2013
 * September 2013
 * August 2013
 * July 2013
 * June 2013
 * May 2013
 * April 2013
 * January 2013
 * December 2012
 * November 2012
 * October 2012
 * September 2012
 * August 2012
 * February 2012
 * January 2012
 * November 2011
 * October 2011
 * August 2011
 * July 2011
 * June 2011
 * May 2011
 * April 2011
 * March 2011
 * February 2011
 * November 2010
 * October 2010

 License


 This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
 Proudly powered by WordPress
Today’s post comes from a recent Go pop quiz. Consider this benchmark fragment. 1  
 func BenchmarkSortStrings(b *testing.B) {
 s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"}
 b.ReportAllocs()
 for i := 0; i &lt; b.N; i++ {
 sort.Strings(s)
 }
} 
A convenience wrapper around sort.Sort(sort.StringSlice(s)), sort.Strings sorts the input in place, so it isn’t expected to allocate (or at least that’s what 43% of the tweeps who responded thought). However it turns out that, at least in recent versions of Go, each iteration of the benchmark causes one heap allocation. Why does this happen?
Interfaces, as all Go programmers should know, are implemented as a two word structure. Each interface value contains a field which holds the type of the interface’s contents, and a pointer to the interface’s contents. 2  
In pseudo Go code, an interface might look something like this:
 type interface struct {
 // the ordinal number for the type of the value
 // assigned to the interface 
 type uintptr
 // (usually) a pointer to the value assigned to
 // the interface
 data uintptr
} 
 interface.data can hold one machine word–8 bytes in most cases–but a []string is 24 bytes; one word for the pointer to the slice’s underlying array; one word for the length; and one for the remaining capacity of the underlying array, so how does Go fit 24 bytes into 8? With the oldest trick in the book, indirection. s, a []string is 24 bytes, but *[]string–a pointer to a slice of strings–is only 8. 
Escaping to the heap
To make the example a little more explicit, here’s the benchmark rewritten without the sort.Strings helper function:
 func BenchmarkSortStrings(b *testing.B) {
 s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"}
 b.ReportAllocs()
 for i := 0; i &lt; b.N; i++ {
 var ss sort.StringSlice = s
 var si sort.Interface = ss // allocation
 sort.Sort(si)
 }
}
 
To make the interface magic work, the compiler rewrites the assignment as var si sort.Interface = &amp;ss–the address of ss is assigned to the interface value. 3  We now have a situation where the interface value holds a pointer to ss, but where does it point? Where in memory does ss live?
It appears that ss is moved to the heap, causing the allocation that the benchmark reports.
 Total: 296.01MB 296.01MB (flat, cum) 99.66%
 8 . . func BenchmarkSortStrings(b *testing.B) { 
 9 . . 	s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"} 
 10 . . 	b.ReportAllocs() 
 11 . . 	for i := 0; i &lt; b.N; i++ { 
 12 . . 	var ss sort.StringSlice = s 
 13 296.01MB 296.01MB 	var si sort.Interface = ss // allocation 
 14 . . 	sort.Sort(si) 
 15 . . 	} 
 16 . . } 
The allocation occurs because the compiler currently cannot convince itself that ss outlives si. The general attitude amongst Go compiler hackers seems to be that this could be improved, but that’s a discussion for another time. As it stands, ss is allocated on the heap. Thus the question becomes, how many bytes are allocated per iteration? Why don’t we ask the testing package.
% go test -bench=. sort_test.go
goos: darwin
goarch: amd64
cpu: Intel(R) Core(TM) i7-5650U CPU @ 2.20GHz
BenchmarkSortStrings-4 12591951 91.36 ns/op 24 B/op 1 allocs/op
PASS
ok command-line-arguments 1.260s
Using Go 1.16beta1, on amd64, 24 bytes are allocated per operation. 4  However, the previous Go version, on the same platform, consumes 32 bytes per operation
% go1.15 test -bench=. sort_test.go
goos: darwin
goarch: amd64
BenchmarkSortStrings-4 11453016 96.4 ns/op 32 B/op 1 allocs/op
PASS
ok command-line-arguments 1.225s
This brings us, circuitously, to the subject of this post, a nifty quality of life improvement coming in Go 1.16. But before I can talk about it, I need to discuss size classes.
Size classes
To explain what a size class is, consider how a theoretical Go runtime could allocate 24 bytes on its heap. A simple way to do this would be keep track of all the memory allocated so far using a pointer to the last allocated byte on the heap. To allocate 24 bytes, the heap pointer is incremented by 24, and the previous value returned to the caller. As long as the code that asked for 24 bytes never writes beyond that mark this mechanism has no overhead. Sadly, in real life, memory allocators don’t just allocate memory, sometimes they have to free it.
Eventually the Go runtime will have to free those 24 bytes, but from the runtime’s point of view, the only thing it knows is the starting address it gave to the caller. It does not know how many bytes after that address were allocated. To permit deallocation, our hypothetical Go runtime allocator would have to record, for each allocation on the heap, its length. Where are the allocation for those lengths allocated? On the heap of course.
In our scenario, when the runtime wants to allocate memory, it could request a little more than it was asked for and use it to store the amount requested. For our slice example, when we asked for 24 bytes, we’d actually consume 24 bytes plus some overhead to store the number 24. How large is this overhead? It turns out the minimum amount that is practical is one word. 5  
To record a 24 byte allocation the overhead would 8 bytes. 25% is not great, but not terrible, and as the size of the allocation goes up, the overhead would become insignificant. However, what would happen if we wanted to store just one byte on the heap? The overhead would be eight times the amount of data we asked for! Is there are more efficient way to allocate small amounts on the heap?
Rather than storing the length alongside each allocation, what if all the thing of the same size were stored together? If all the 24 byte things are stored together, then the runtime would automatically know how large they are. All the runtime needs is a single bit to indicate if a 24 byte area is in use or not. In Go these areas are known as size classes, because all the things of the same size are stored together (think school class–all the students are the same grade–not a C++ class). When the runtime needs to allocate a small amount, it does so using the smallest size class that can accomodate the allocation.
Unlimited size classes for all
Now we know how size classes work, the obvious question is, where are they stored? Not surprisingly, the memory for a size class comes from the heap. To minimise overhead, the runtime allocates a larger amount from the heap (usually a multiple of the system page size) then dedicates that space for allocations of a single size. But, there’s a problem.
The pattern of allocating a large area to store things of the same size works well 6  if there’s a fixed, preferably small, number of allocation sizes, but in a general purpose language programs can ask the runtime for an allocation of any size. 7  
For example, imagine asking the runtime for 9 bytes. 9 bytes is an uncommon size, so it’s likely that a new size class for things of size 9 would be required. As 9 byte things are uncommon, it’s likely the remainder of the allocation, 4kb or more, would be wasted. Because of this the set of possible size classes is fixed. If a size class of the exact amount is not available, the allocation is rounded up to the next size class. In our example 9 bytes might be allocated in the 12 byte size class. The overhead of 3 unused bytes is better than an entire size class allocation which goes mostly unused.
All together now
This is the final piece of the puzzle. Go 1.15 did not have a 24 byte size class, so the heap allocation of ss was allocated in the 32 byte size class. Thanks to the work of Martin Möhrmann Go 1.16 has a 24 byte size class, which is a perfect fit for slice values assigned to interfaces.
	
	
	
	
	
Picture yourself, an engineer working at the hottest distributed microservices de jour, assigned to fix a bug. You jump into an unfamiliar codebase and quickly locate the line where the problem occurred. The fix is simple, just return early or substitute a default value in the case that one cannot be determined from your input. Boom, problem solved. Code compiles, fix goes out to production, and the card’s in the done pile in time for Tuesday drinks at the pub.
Now consider this alternative scenario. You find the problematic line and, before you make the fix, you decide to add a test so you will know that your fix worked, and hopefully someone will not accidentally revert it in the future.
You’ve figured out that some weird edge case causes that line to be called without being able to determine the right value for y. You can see it clearly, y ends up being zero so the program crashes. To write a test, you need to get the conditions that caused y to be zero to occur on command. There are only two parameters passed into this function, this shouldn’t be hard. Oh, but this is a method, not a function, so you need to construct an instance of the object in the right state to trigger the bug. Hmm, this language uses constructors to make sure people can’t just monkey up the bits of the object they need for a test, you’ll have to find, mock, stub or build instances of all the objects dependencies (and their dependencies, and so on) then run the object through the precise series of operations to put the it in the state that causes y to be empty. Then you can write the test.
At this point Tuesday night drinks are fading before you eyes. Without being able to write a test for your fix, how will you convince a reviewer that it actually fixed what it fixed? All you can prove was your “simple” change didn’t break any behaviour that was covered by existing tests. 
Fast forward to Friday and your one line change has finally been merged, along with a few new test cases to make sure it doesn’t happen again. Next week you’re going to sit down and try to figure out a bunch of weird crashes you caused while trying to stub out calls to the database. Eventually you gave up and copied some code from another test that uses a copy of production data. It’s much slower than it needs to be, but it worked, and was the only way to put a reasonable estimate on a job that had already consumed your entire week.
The moral of the story; was the test the reason it took nearly a week to make a one line fix?
	
	
	
	
	
The Go compiler’s SSA backend contains a facility to produce HTML debugging output of the compilation phases. This post covers how to print the SSA output for function and methods.
Let’s start with a sample program which contains a function, a value method, and a pointer method:
 package main
import (
 "fmt"
)
type Numbers struct {
 vals []int
}
func (n *Numbers) Add(v int) {
 n.vals = append(n.vals, v)
}
func (n Numbers) Average() float64 {
 sum := 0.0
 for _, num := range n.vals {
 sum += float64(num)
 }
 return sum / float64(len(n.vals))
}
func main() {
 var numbers Numbers
 numbers.Add(200)
 numbers.Add(43)
 numbers.Add(-6)
 fmt.Println(numbers.Average())
} 
Control of the SSA debugging output is via the GOSSAFUNC environment variable. This variable takes the name of the function to dump. This is not the functions fully qualified name. For func main above the name of the function is main not main.main.
% env GOSSAFUNC=main go build
runtime
dumped SSA to ../../go/src/runtime/ssa.html
t
dumped SSA to ./ssa.html
In this example GOSSAFUNC=main matched both main.main and a function called runtime.main. 1  This is a little unfortunate, but in practice probably not a big deal as, if you’re performance tuning your code, it won’t be in a giant spaghetti blob in func main. 
What is more likely is your code will be in a method, so you’ve probably landed on this post looking for the correct incantation to dump the SSA output for a method.
To print the SSA debug for the pointer method func (n *Numbers) Add, the equivalent function name is(*Numbers).Add: 2  
% env "GOSSAFUNC=(*Numbers).Add" go build
t
dumped SSA to ./ssa.html
To print the SSA debug for a value method func (n Numbers) Average, the equivalent function name is (*Numbers).Average even though this is a value method:
% env "GOSSAFUNC=(*Numbers).Average" go build
t
dumped SSA to ./ssa.html
	
	
	
	
	
Per the overlapping interfaces proposal, Go 1.14 now permits embedding of interfaces with overlapping method sets. This is a brief post explain what this change means:
Let’s start with the definition of the three key interfaces from the io package; io.Reader, io.Writer, and io.Closer:
 package io
type Reader interface {
 Read([]byte) (int, error)
}
type Writer interface {
 Write([]byte) (int, error)
}
type Closer interface {
 Close() error
}  
Just as embedding a type inside a struct allows the embedded type’s fields and methods to be accessed as if it were declared on the embedding type 1 , the process is true for interfaces. Thus there is no difference between explicitly declaring
 type ReadCloser interface {
 Read([]byte) (int, error)
 Close() error
} 
and using embedding to compose the interface
 type ReadCloser interface {
 Reader
 Closer
} 
You can even mix and match 
 type WriteCloser interface {
 Write([]byte) (int, error)
 Closer
} 
However, prior to Go 1.14, if you continued to compose interface declarations in this manner you would likely find that something like this,
 type ReadWriteCloser interface {
 ReadCloser
 WriterCloser
} 
would fail to compile
% go build interfaces.go
command-line-arguments
./interfaces.go:27:2: duplicate method Close
Fortunately, with Go 1.14 this is no longer a limitation, thus solving problems that typically occur with diamond-shaped embedding graphs.
However, there is a catch that I ran into attempting to demonstrate this feature to the local user group–this feature is only enabled when the Go compiler uses the 1.14 (or later) spec.
As near as I can make out the rules for which version of the Go spec is used during compilation appear to be:
 If your source code is stored inside GOPATH (or you have disabled modules with GO111MODULE=off) then the version of the Go spec used to compile with matches the version of the compiler you are using. Said another way, if you have Go 1.13 installed, your Go version is 1.13. If you have Go 1.14 installed, your version is 1.14. No surprises here. If your source code is stored outside GOPATH (or you have forced modules on with GO111MODULE=on) then the go tool will take the Go version from the go.mod file.  If there is no Go version listed in go.mod then the version of the spec will be the version of Go installed. This is identical to point 1. If you are in module mode, either by being outside GOPATH or with GO111MODULE=on, but there is no go.mod file in the current, or any parent, directory then the version of the Go spec used to compile your code defaults to Go 1.13. 
The last point caught me out.
	
	
	
	
	
A few days ago Fatih posted this question on twitter.
I’m going to attempt to give my answer, however to do that I need to apply some simplifications as my previous attempts to answer it involved a lot of phrases like a pointer to a pointer, and other unhelpful waffling. Hopefully my simplified answer can be useful in building a mental framework to answer Fatih’s original question.
Restating the question
Fatih’s original tweet showed four different variations of json.Unmarshal. I’m going to focus on the last two, which I’ll rewrite a little:
 package main
import (
 "encoding/json"
 "fmt"
)
type Result struct {
 Foo string `json:"foo"`
}
func main() {
 content := []byte(`{"foo": "bar"}`)
 var result1, result2 *Result
 err := json.Unmarshal(content, &amp;result1)
 fmt.Println(result1, err) // &amp;{bar} &lt;nil&gt;
 err = json.Unmarshal(content, result2)
 fmt.Println(result2, err) // &lt;nil&gt; json: Unmarshal(nil *main.Result)
} 
Restated in words, result1 and result2 are the same type; *Result. Decoding into result1 works as expected, whereas decoding into result2 causes the json package to complain that the value passed to Unmarshal is nil. However, both values were declared without an initialiser so both would have taken on the type’s zero value, nil.
Eagle eyed readers will have spotted that the reason for the difference is the first invocation is passed &amp;result1, while the second is passed result2, but this explanation is unsatisfactory because the documentation for json.Unmarshal states:
 Unmarshal parses the JSON-encoded data and stores the result in the value pointed to by v. If v is nil or not a pointer, Unmarshal returns an InvalidUnmarshalError. 
Which is confusing because result1 and result2 are pointers. Furthermore, without initialisation, both are nil. Now, the documentation is correct (as you’d expect from a package that has been hammered on for a decade), but explaining why takes a little more investigation.
Functions receive a copy of their arguments
Every assignment in Go is a copy, this includes function arguments and return values.
 package main
import (
 "fmt"
)
func increment(v int) {
 v++
}
func main() {
 v := 1
 increment(v)
 fmt.Println(v) // 1
} 
In this example, increment is operating on a copy of main‘s v. This is because the v declared in main and increment‘s v parameter have different addresses in memory. Thus changes to increment‘s v cannot affect the contents of main‘s v.
 package main
import (
 "fmt"
)
func increment(v *int) {
 *v++
}
func main() {
 v := 1
 increment(&amp;v)
 fmt.Println(v) // 2
} 
If we wanted to write increment in a way that it could affect the contents of its caller we would need to pass a reference, a pointer, to main.v. 1  This example demonstrates why json.Unmarshal needs a pointer to the value to decode JSON into.
Pointers to pointers
Returning to the original question, both result1 and result2 are declared as *Result, that is, pointers to a Result value. We established that you have to pass the address of caller’s value to json.Unmarshal otherwise it won’t be able to alter the contents of the caller’s value. Why then must we pass the address of result1, a **Result, a pointer to a pointer to a Result, for the operation to succeed.
To explain this another detour is required. Consider this code:
 package main
import (
 "encoding/json"
 "fmt"
)
type Result struct {
 Foo *string `json:"foo"`
}
func main() {
 content := []byte(`{"foo": "bar"}`)
 var result1 *Result
 err := json.Unmarshal(content, &amp;result1)
 fmt.Printf("%#v %v", result1, err) // &amp;main.Result{Foo:(*string)(0xc0000102f0)} &lt;nil&gt;
} 
In this example Result contains a pointer typed field, Foo *string. During JSON decoding Unmarshal allocated a new string value, stored the value bar in it, then placed the address of the string in Result.Foo. This behaviour is quite handy as it frees the caller from having to initialise Result.Foo and makes it easier to detect when a field was not initialised because the JSON did not contain a value. Beyond the convenience this offers for simple examples it would be prohibitively difficult for the caller to properly initialise all the reference type fields in a structure before decoding unknown JSON without first inspecting the incoming JSON which itself may be problematic if the input is coming from an io.Reader without the ability to rewind the input.
 To unmarshal JSON into a pointer, Unmarshal first handles the case of the JSON being the JSON literal null. In that case, Unmarshal sets the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the value pointed at by the pointer. If the pointer is nil, Unmarshal allocates a new value for it to point to. 
 json.Unmarshal‘s handling of pointer fields is clearly documented, and works as you would expect, allocating a new value whenever there is a need to decode into a pointer shaped field. It is this behaviour that gives us a hint to what is happening in the original example.
We’ve seen that when json.Unmarshal encounters a field which points to nil it will allocate a new value of the correct type and assign its address the field before proceeding. Not only is does behaviour is applied recursively–for example in the case of a complex structure which contains pointers to other structures–but it also applies to the value passed to Unmarshal. 
 package main
import (
 "encoding/json"
 "fmt"
)
func main() {
 content := []byte(`1`)
 var result *int
 err := json.Unmarshal(content, &amp;result)
 fmt.Println(*result, err) // 1 &lt;nil&gt;
} 
In this example result is not a struct, but a simple *int which, lacking an initialiser, defaults to nil. After JSON decoding, result now points to an int with the value 1.
Putting the pieces together
Now I think I’m ready to take a shot at answering Fatih’s question.
 json.Unmarshal requires the address of the variable you want to decode into, otherwise it would decode into a temporary copy which would be discard on return. Normally this is done by declaring a value, then passing its address, or explicitly initialising the the value
 var result1 Result
err := json.Unmarshal(content, &amp;result1) // this is fine
var result2 = new(Result)
err = json.Unmarshal(content, result2) // and this
var result3 = &amp;Result{}
err = json.Unmarshal(content, result3) // this is also fine 
In all three cases the address that the *Result points too is not nil, it points to initialised memory that json.Unmarshal decodes into.
Now consider what happens when json.Unmarshal encounters this
 var result4 *Result
err = json.Unmarshal(content, result4) // err json: Unmarshal(nil *main.Result) 
 result2, result3, and the expression &amp;result1 point to a Result. However result4, even though it has the same type as the previous three, does not point to initialised memory, it points to nil. Thus, according to the examples we saw previously, before json.Unmarshal can decode into it, the memory result4 points too must be initialised. 
However, because each function receives a copy of its arguments, the caller’s result4 variable and the copy inside json.Unmarshal are unique. json.Unmarshal can allocate a new Result value and decode into it, but it cannot alter result4 to point to this new value because it was not provided with a reference to result4, only a copy of its contents.
	
	
	
	
	
Conventional wisdom dictates that the larger the number of types declared in a Go program, the larger the resulting binary. Intuitively this makes sense, after all, what’s the point in defining a bunch of types if you’re not going to write code that operates on them. However, part of the job of a linker is to detect functions which are not referenced by a program–say they are part of a library of which only a subset of functionality is used–and remove them from the final output. Yet, the adage mo’ types, mo’ binary holds true for the majority of Go programs.
In this post I’ll dig into what equality, in the context of a Go program, means and why changes like this have a measurable impact on the size of a Go program.
Defining equality between two values
The Go spec defines the concepts of assignability and equality. Assignabiity is the act of assigning a value to an identifier. Not everything which is declared can be assigned, for example constants and functions. Equality is the act of comparing two identifies by asking are their contents the same? 
Being a strongly typed language, the notion of sameness is fundamentally rooted in the identifier’s type. Two things can only be the same if they are of the same type. Beyond that, the type of the values defines how they are compared.
For example, integers are compared arithmetically. For pointer types, equality is determining if the addresses they point too are the same. Reference types like maps and channels, like pointers, are considered to be the same if they have the same address.
These are all examples of bitwise equality, that is, if the bit patterns of the memory that value occupies are the same, those values are equal. This is known as memcmp, short for memory comparison, as equality is defined by comparing the contents of two areas of memory.
Hold on to this idea, I’ll come back to in a second.
Struct equality
Beyond scalar types like integers, floats, and pointers is the realm of compound types; structs. All structs are laid out in memory in program order, thus this declaration:
 type S struct {
 a, b, c, d int64
} 
will consume 32 bytes of memory; 8 bytes for a, then 8 bytes for b, and so on. The spec says that struct values are comparable if all their fields are comparable. Thus two structs are equal iff each of their fields are equal.
 a := S{1, 2, 3, 4}
b := S{1, 2, 3, 4}
fmt.Println(a == b) // prints true 
Under the hood the compiler uses memcmp to compare the 32 bytes of a and b.
Padding and alignment
However the simplistic bitwise comparison strategy will fail in situations like this:
 type S struct {
 a byte
 b uint64
 c int16
 d uint32
}
func main()
 a := S{1, 2, 3, 4}
 b := S{1, 2, 3, 4}
 fmt.Println(a == b) // prints true
} 
The code compiles, the comparison is still true, but under the hood the compiler cannot rely on comparing the bit patterns of a and b because the structure contains padding.
Go requires each field in a struct to be naturally aligned. 2 byte values must start on an even address, four byte values on an address divisible by 4, and so on 1 . The compiler inserts padding to ensure the fields are aligned to according to their type and the underlying platform. In effect, after padding, this is what the compiler sees 2 :
 type S struct {
 a byte
 _ [7]byte // padding
 b uint64
 c int16
 _ [2]int16 // padding
 d uint32
} 
Padding exists to ensure the correct field alignments, and while it does take up space in memory, the contents of those padding bytes are unknown. You might assume that, being Go, the padding bytes are always zero, but it turns out that’s not the case–the contents of padding bytes are simply not defined. Because they’re not defined to always be a certain value, doing a bitwise comparison may return false because the nine bytes of padding spread throughout the 24 bytes of S are may not be the same.
The Go compiler solves this problem by generating what is known as an equality function. In this case S‘s equality function knows how to compare two values of type S by comparing only the fields in the function while skipping over the padding.
Type algorithms
Phew, that was a lot of setup to illustrate why, for each type defined in a Go program, the compiler may generate several supporting functions, known inside the compiler as the type’s algorithms. In addition to the equality function the compiler will generate a hash function if the type is used as a map key. Like the equality function, the hash function must consider factors like padding when computing its result to ensure it remains stable. 
It turns out that it can be hard, and sometimes non obvious, to intuit when the compiler will generate these functions–it’s more than you’d expect–and it can be hard for the linker to eliminate the ones that are not needed as reflection often causes the linker to be more conservative when trimming types.
Reducing binary size by prohibiting comparisons
Now we’re at a point to explain Brad’s change. By adding an incomparable field  3  to the type, the resulting struct is by extension incomparable, thus forcing the compiler to elide the generation of eq and hash algorithms, short circuiting the linkers elimination of those types and, in practice, reducing the size of the final binary. As an example of this technique, this program:
 package main
import "fmt"
func main() {
 type t struct {
 // _ [0][]byte uncomment to prevent comparison
 a byte
 b uint16
 c int32
 d uint64
 }
 var a t
 fmt.Println(a)
} 
when compiled with Go 1.14.2 (darwin/amd64), decreased from 2174088 to 2174056, a saving of 32 bytes. In isolation this 32 byte saving may seem like small beer, but consider that equality and hash functions can be generated for every type in the transitive closure of your program and all its dependencies, and the size of these functions varies depending on the size of the type and its complexity, prohibiting them can have a sizeable impact on the final binary over and above the old saw of -ldflags="-s -w".
The bottom line, if you don’t wish to make your types comparable, a hack like this enforces it at the source level while contributing to a small reduction in the size of your binary.
Addendum: thanks to Brad’s prodding, Go 1.15 already has a bunch of improvements by Cherry Zhang and Keith Randall that fix the most egregious of the failures to eliminate unnecessary equality and hash functions (although I suspect it was also to avoid the proliferation of this class of CLs).
	
	
	
	
	
In the previous post I discussed how leaf inlining allows the Go compiler to reduce the overhead of function calls and extend optimisation opportunities across function boundaries. In this post I’ll discuss the limits of inlining and leaf vs mid-stack inlining.
The limits of inlining
Inlining a function into its caller removes the call’s overhead and increases the opportunity for the compiler to apply additional optimisations so the question should be asked, if some inlining is good, would more be better, why not inline as much as possible? 
Inlining trades possibly larger program sizes for potentially faster execution time. The main reason to limit inlining is creating many inlined copies of a function can increase compile time and result in larger binaries for marginal gain. Even taking into account the opportunities for further optimisation, aggressive inlining tends to increase the size of, and the time too compile, the resulting binary.
Inlining works best for small functions that do relatively little work compared to the overhead of calling them. As the size of a function grows, the time saved avoiding the call’s overhead diminishes relative to the work done inside the function. Larger functions tend to be more complex, thus the benefits of optimising their inlined forms vs in situ are reduced.
Inlining budget
During compilation each function’s inlineabilty is calculated using what is known as the inlining budget  1 . The cost calculation can be tricky to internalise but is broadly one unit per node in the AST for simple things like unary and binary operations but can be higher for complex operations like make. Consider this example:
 package main
func small() string {
 s := "hello, " + "world!"
 return s
}
func large() string {
 s := "a"
 s += "b"
 s += "c"
 s += "d"
 s += "e"
 s += "f"
 s += "g"
 s += "h"
 s += "i"
 s += "j"
 s += "k"
 s += "l"
 s += "m"
 s += "n"
 s += "o"
 s += "p"
 s += "q"
 s += "r"
 s += "s"
 s += "t"
 s += "u"
 s += "v"
 s += "w"
 s += "x"
 s += "y"
 s += "z"
 return s
}
func main() {
 small()
 large()
} 
Compiling this function with -gcflags=-m=2 allows us to see the cost the compiler assigns to each function.
% go build -gcflags=-m=2 inl.go 
# command-line-arguments
./inl.go:3:6: can inline small with cost 7 as: func() string { s := "hello, world!"; return s }
./inl.go:8:6: cannot inline large: function too complex: cost 82 exceeds budget 80
./inl.go:38:6: can inline main with cost 68 as: func() { small(); large() }
./inl.go:39:7: inlining call to small func() string { s := "hello, world!"; return s }
The compiler determined that func small() can be inlined due to its cost of 7. func large() was determined to be too expensive. func main()has been marked as eligible and assigned a cost of 68; 7 from the body of small, 57 from the function call to small and the remainder in its own overhead.
The inlining budget can be controlled to some degree with the -gcflag=-l flag. Currently the values that apply are:
  -gcflags=-l=0 is the default level of inlining.  -gcflags=-l (or -gcflags=-l=1) disables inlining.  -gcflags=-l=2 and -gcflags=-l=3 are currently unused and have no effect over -gcflags=-l=0   -gcflags=-l=4 reduces the cost for inlining non-leaf functions and calls through interfaces. 2   
Hairy optimisations
Some functions with a relatively low inlining cost may be ineligible because of their complexity. This is known as the function’s hairiness as the semantics of some operations are hard to reason about once inlined, for example recover, break. Others, like select and go, involve co-ordination with the runtime so the extra effort of inlining doesn’t pay for itself.
The list of hairy statements also includes things like for and range which don’t have an inherently large cost, but simply haven’t been optimised yet.
Mid stack inlining
Historically the Go compiler only performed leaf inlining–only functions which did not call other functions were eligible. In the context of the hairiness discussion previously, a function call would disqualify the function from being inlined.
Enter mid stack inlining which, as its name implies, allows functions in the middle of a call stack to be inlined without requiring everything below them to be eligible. Mid stack inlining was introduced by David Lazar in Go 1.9 and improved in subsequent releases. This presentation goes into some of the difficulties with retaining the behaviour of stack traces and runtime.Callers in code paths that had been heavily inlined.
We see an example of mid-stack inlining in the previous example. After inlining, func main() contains the body of func small() and a call to func large(), thus it is considered a non-leaf function. Historically this would have prevented it from being further inlined even though its combined cost was less than the inlining budget.
The primary use case for mid stack inlining is to reduce the overhead of a path through the call stack. Consider this example:
 package main
import (
 "fmt"
 "strconv"
)
type Rectangle struct {}
//go:noinline
func (r *Rectangle) Height() int {
 h, _ := strconv.ParseInt("7", 10, 0)
 return int(h)
}
func (r *Rectangle) Width() int {
 return 6
}
func (r *Rectangle) Area() int { return r.Height() * r.Width() }
func main() {
 var r Rectangle
 fmt.Println(r.Area())
} 
In this example r.Area() is a simple function which calls two others. r.Width() can be inlined while r.Height(), simulated here with the //go:noinline annotation, cannot.  3  
% go build -gcflags='-m=2' square.go 
# command-line-arguments
./square.go:12:6: cannot inline (*Rectangle).Height: marked go:noinline 
./square.go:17:6: can inline (*Rectangle).Width with cost 2 as: method(*Rectangle) func() int { return 6 }
./square.go:21:6: can inline (*Rectangle).Area with cost 67 as: method(*Rectangle) func() int { return r.Height() * r.Width() } ./square.go:21:61: inlining call to (*Rectangle).Width method(*Rectangle) func() int { return 6 } 
./square.go:23:6: cannot inline main: function too complex: cost 150 exceeds budget 80 
./square.go:25:20: inlining call to (*Rectangle).Area method(*Rectangle) func() int { return r.Height() * r.Width() }
./square.go:25:20: inlining call to (*Rectangle).Width method(*Rectangle) func() int { return 6 }
As the multiplication performed by r.Area() is cheap compared to the overhead of calling it, inlining r.Area()‘s single expression is a net win even if its downstream caller to r.Height() remains ineligible.
Fast path inlining
The most startling example of the power of mid-stack inlining comes from 2019 when Carlo Alberto Ferraris improved the performance of sync.Mutex.Lock() by allowing the fast path of the lock–the uncontended case–to be inlined into its caller. Prior to this change sync.Mutex.Lock() was a large function containing many hairy conditions which made it ineligible to be inlined. Even in the case where the lock was available, the caller had to pay the overhead of calling sync.Mutex.Lock().
Carlo’s change split sync.Mutex.Lock() into two functions (a process he dubbed outlining). The outer sync.Mutex.Lock() method now calls sync/atomic.CompareAndSwapInt32() and returns to the caller immediately if the CAS succeeds. If not, the function falls through to sync.Mutex.lockSlow() which handles the slow path required to register interest on the lock and park the goroutine. 4  
% go build -gcflags='-m=2 -l=0' sync 2&gt;&amp;1 | grep '(*Mutex).Lock'
../go/src/sync/mutex.go:72:6: can inline (*Mutex).Lock with cost 69 as: method(*Mutex) func() { if "sync/atomic".CompareAndSwapInt32(&amp;m.state, 0, mutexLocked) { if race.Enabled {  }; return  }; m.lockSlow() }
By splitting the function into an easily inlineable outer function, falling through to a complex inner function to handle the slow path Carlo’s combined mid stack inlining and the compiler’s support for intrinsic operations to reduce the cost of an uncontended lock by 14%. Then he repeated the trick for an additional 9% saving in  sync.RWMutex.Unlock() .
	
	
	
	
	
This is a post about how the Go compiler implements inlining and how this optimisation affects your Go code. 
 n.b. This article focuses on gc, the de facto Go compiler from golang.org. The concepts discussed apply broadly to other Go compilers like gccgo and tinygo but may differ in implementation and efficacy.
What is inlining?
Inlining is the act of combining smaller functions into their respective callers. In the early days of computing this optimisation was typically performed by hand. Nowadays inlining is one of a class of fundamental optimisations performed automatically during the compilation process.
Why is inlining important?
Inlining is important for two reasons. The first is it removes the overhead of the function call itself. The second is it permits the compiler to more effectively apply other optimisation strategies.
Function call overhead
Calling a function 1  in any language carries a cost. There are the overheads of marshalling parameters into registers or onto the stack (depending on the ABI) and reversing the process on return. Invoking a function call involves jumping the program counter from one point in the instruction stream to another which can cause a pipeline stall. Once inside the function there is usually some preamble required to prepare a new stack frame for the function to execute and a similar epilogue needed to retire the frame before returning to the caller.
In Go a function call carries additional costs to support dynamic stack growth. On entry the amount of stack space available to the goroutine is compared to the amount required for the function. If insufficient stack space is available, the preamble jumps into the runtime logic that grows the stack by copying it to a new, larger, location. Once this is done the runtime jumps back to the start of the original function, the stack check is performed again, which now passes, and the call continues. In this way goroutines can start with a small stack allocation which grows only when needed. 2  
This check is cheap–only a few instructions–and because goroutine stacks grows geometrically the check rarely fails. Thus, the branch prediction unit inside a modern processor can hide the cost of the stack check by assuming it will always be successful. In the case where the processor mis-predicts the stack check and has to discard the work done while it was executing speculatively, the cost of the pipeline stall is relatively small compared to the cost of the work needed for the runtime to grow a goroutines stack.
While the overhead of the generic and Go specific components of each function call are well optimised by modern processors using speculative execution techniques, those overheads cannot be entirely eliminated, thus each function call carries with it a performance cost over and above the time it takes to perform useful work. As a function call’s overhead is fixed, smaller functions pay a larger cost relative to larger ones because they tend to do less useful work per invocation.
The solution to eliminating these overheads must therefore be to eliminate the function call itself, which the Go compiler does, under certain conditions, by replacing the call to a function with the contents of the function. This is known as inlining because it brings the body of the function in line with its caller.
Improved optimisation opportunities
Dr. Cliff Click describes inlining as the optimisation performed by modern compilers as it forms the basis for optimisations like constant propagation and dead code elimination. In effect, inlining allows the compiler to see  further, allowing it to observe, in the context that a particular function is being called, logic that can be further simplified or eliminated entirely. As inlining can be applied recursively optimisation decisions can be made not only in the context of each individual function, but also applied to the chain of functions in a call path.
Inlining in action
The effects of inlining can be demonstrated with this small example
 package main
import "testing"
//go:noinline
func max(a, b int) int {
 if a &gt; b {
 return a
 }
 return b
}
var Result int
func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i &lt; b.N; i++ {
 r = max(-1, i)
 }
 Result = r
} 
Running this benchmark gives the following result: 3  
% go test -bench=. 
BenchmarkMax-4   530687617         2.24 ns/op
The cost of max(-1, i) is around 2.24 nanoseconds on my 2015 MacBook Air. Now let’s remove the //go:noinline pragma and see the result:
% go test -bench=. 
BenchmarkMax-4   1000000000         0.514 ns/op
From 2.24 ns to 0.51 ns, or according to benchstat, a 78% improvement.
% benchstat {old,new}.txt
name   old time/op  new time/op  delta
Max-4  2.21ns ± 1%  0.49ns ± 6%  -77.96%  (p=0.000 n=18+19)
Where did these improvements come from? 
First, the removal of the function call and associated preamble 4  was a major contributor. Pulling the contents of max into its caller reduced the number of instructions executed by the processor and eliminated several branches.
Now the contents of max are visible to the compiler as it optimises BenchmarkMax it can make some additional improvements. Consider that once max is inlined, this is what the body of BenchmarkMax looks like to the compiler:
 func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i &lt; b.N; i++ {
 if -1 &gt; i {
 r = -1
 } else {
 r = i
 }
 }
 Result = r
} 
Running the benchmark again we see our manually inlined version performs as well as the version inlined by the compiler
% benchstat {old,new}.txt
name   old time/op  new time/op  delta
Max-4  2.21ns ± 1%  0.48ns ± 3%  -78.14%  (p=0.000 n=18+18)
Now the compiler has access to the result of inlining max into BenchmarkMax it can apply optimisation passes which were not possible before. For example, the compiler has noted that i is initialised to 0 and only incremented so any comparison with i can assume i will never be negative. Thus, the condition -1 &gt; i will never be true. 5  
Having proved that -1 &gt; i will never be true, the compiler can simplify the code to
 func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i &lt; b.N; i++ {
 if false {
 r = -1
 } else {
 r = i
 }
 }
 Result = r
} 
and because the branch is now a constant, the compiler can eliminate the unreachable path leaving it with
 func BenchmarkMax(b *testing.B) {
 var r int
 for i := 0; i &lt; b.N; i++ {
 r = i
 }
 Result = r
} 
Thus, through inlining and the optimisations it unlocks, the compiler has reduced the expression r = max(-1, i) to simply r = i.
The limits of inlining
In this article I’ve discussed, so called, leaf inlining; the act of inlining a function at the bottom of a call stack into its direct caller. Inlining is a recursive process, once a function has been inlined into its caller, the compiler may inline the resulting code into its caller, as so on. For example, this code 
 func BenchmarkMaxMaxMax(b *testing.B) {
 var r int
 for i := 0; i &lt; b.N; i++ {
 r = max(max(-1, i), max(0, i))
 }
 Result = r
} 
runs as fast as the previous example as the compiler is able to repeatedly apply the optimisations outlined above to reduce the code to the same r = i expression.
In the next article I’ll discuss an alternative inlining strategy when the Go compiler wishes to inline a function in the middle of a call stack. Finally I’ll discuss the limits that the compiler is prepared to go to to inline code, and which Go constructs are currently beyond its capability.
	
	
	
	
	
The testing package is one of my favourite packages in the Go standard library, not just because of its low noise approach to unit testing, but, over the lifetime of Go, it has received a steady stream of quality of life improvements driven by real world usage.
The most recent example of this is, in Go 1.14, go test -v will stream t.Log output as it happens, rather than hoarding it til the end of the test run. Here’s an example;
 package main
import (
	"fmt"
	"testing"
	"time"
)
func TestLogStreaming(t *testing.T) {
	for i := 0; i &lt; 5; i++ {
	time.Sleep(300 * time.Millisecond)
	fmt.Println("fmt.Println:", i)
	t.Log("t.Log:", i)
	}
} 
 Note: Calling fmt.Println inside a test is generally considered a no no as it bypasses the testing package’s output buffering irrespective of the -v flag. However, for this example, it‘s necessary to demonstrate the streaming t.Log change.
 % go1.13 test -v tlog_test.go
=== RUN TestLogStreaming
fmt.Println: 0
fmt.Println: 1
fmt.Println: 2
fmt.Println: 3
fmt.Println: 4
--- PASS: TestLogStreaming (1.52s)
 tlog_test.go:13: t.Log: 0
 tlog_test.go:13: t.Log: 1
 tlog_test.go:13: t.Log: 2
 tlog_test.go:13: t.Log: 3
 tlog_test.go:13: t.Log: 4
PASS
ok command-line-arguments 1.971s 
Under Go 1.13 and earlier the fmt.Println lines output immediately. t.Log lines are buffered and are printed after the test completes.
 % go1.14 test -v tlog_test.go
=== RUN TestLogStreaming
fmt.Println: 0
 TestLogStreaming: tlog_test.go:13: t.Log: 0
fmt.Println: 1
 TestLogStreaming: tlog_test.go:13: t.Log: 1
fmt.Println: 2
 TestLogStreaming: tlog_test.go:13: t.Log: 2
fmt.Println: 3
 TestLogStreaming: tlog_test.go:13: t.Log: 3
fmt.Println: 4
 TestLogStreaming: tlog_test.go:13: t.Log: 4
--- PASS: TestLogStreaming (1.51s)
PASS
ok command-line-arguments 1.809s 
Under Go 1.14 the fmt.Println and t.Log lines are interleaved, rather than waiting for the test to complete, demonstrating that test output is streamed when go test -v is used.
This is a great quality of life improvement for integration style tests that often retry for long periods when the test is failing. Streaming t.Log output will help Gophers debug those test failures without having to wait until the entire test times out to receive their output. 
	
	
	
	
	
Programmers have a tendency to be superstitious. Particularly, when a programmer hears that copies are expensive, they start to see them everywhere, especially when they learn that, in Go, every assignment is a copy.
Consider this code; x is three orders of magnitude larger than y, is the assignment of x to a more expensive than the assignment of y to b?
func f() {
 x, y := make([]byte, 9000), make([]byte, 9)
 a := x
 b := y
 // ...
 } 
The answer is; no. x and y have the same type, []byte, that is, a slice of bytes. As both variables have the same type, their assignment involves copying the same amount of data. Both assignments have the same cost.
All slices are the same size; three machine words (three uintptrs). The first word in the slice is a pointer to the slice’s backing array, the storage for the slice, the second word is the slice’s length, and the third is the capacity. Assigning one slice variable to another copies just three machine words. 
 Further reading:  Go slices: usage and internals (blog.golang.org)
A few bytes here, a few there, pretty soon you’re talking real memory
https://dave.cheney.net/
https://princiya.com/blog/
P`s Blog
2019-04-23
None
<div><div><main><p></p><div><article><div><header><h3><a href="https://medium.com/free-code-camp/how-to-connect-react-to-redux-a-diagrammatic-guide-d2687c14750a">How to connect react to redux - a diagrammatic guide</a></h3><small>April 23, 2019</small></header><p></p><p>Here is a diagrammatic guide to better understand how Redux connects to React.</p></div></article><article><div><header><h3><a href="https://medium.com/free-code-camp/what-you-should-know-about-web-tracking-and-how-it-affects-your-online-privacy-42935355525">Web Tracking - what you should know about your privacy online</a></h3><small>April 23, 2018</small></header><p></p><p>This is my attempt to explain how web tracking works in a nutshell.</p></div></article><article><div><header><h3><a href="https://princiya777.wordpress.com/2017/08/06/webrtc-a-detailed-history/">WebRTC - a detailed history</a></h3><small>August 06, 2017</small></header><p></p><p>The initial version of the OpenWebRTC implementation was developed internally at Ericsson Research. The latter is maintained by the Google Chrome team.</p></div></article></div><div><article><div><header><h3><a href="https://princiya.com/blog/docker-k8s-for-frontend/">Is Docker, Kubernetes necessary for frontend development?</a></h3><small>January 7, 2022 • ☕️ 3 min read</small></header><p>I am an aspiring Front end developer. Currently doing self study on Html, Css, JavaScript and React. I've a question - as a frontend developer do you use Linux, docker, kubernetes?</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/serverless-react/">Serverless React</a></h3><small>January 4, 2022 • ☕️ 2 min read</small></header><p>Serverless React application deployed using Netlify, and an overview of databases to make serverless functions stateful.</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/how-to-learn/">How to learn</a></h3><small>December 30, 2021 • ☕️ 3 min read</small></header><p>Tips on how to learn new things.</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/thankful/">Thankful and grateful</a></h3><small>November 27, 2021 • ☕️ 3 min read</small></header><p>There is always something to be thankful for</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/how-to-get-first-tech-job/">How to get the first tech job</a></h3><small>August 24, 2021 • ☕️ 1 min read</small></header><p>Resources to help you find your first tech job</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/salary-negotiation-part-2/">Salary negotiation tips for Software Engineers</a></h3><small>May 14, 2021 • ☕️ 6 min read</small></header><p>3 mistakes to avoid and tips to ace at your next salary negotiation</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/salary-negotiation-part-1/">What should you know about salary negotiation?</a></h3><small>May 12, 2021 • ☕️ 5 min read</small></header><p>How to prepare for your next salary negotiation by knowing what to negotiate</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/mlh-april-2021/">MLH Graduation 2021</a></h3><small>April 16, 2021 • ☕️ 1 min read</small></header><p>Major league hacking graduation speech to the April 2021 graduating batch</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/finding-my-focus/">Finding my focus</a></h3><small>March 21, 2021 • ☕️ 3 min read</small></header><p>Getting started with the Blogging for Devs Spring Accelerator</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/gif-for-social-card-images-gatsby/">How to create automated GIF social sharing cards</a></h3><small>January 8, 2021 • ☕️ 3 min read</small></header><p></p><p>A 3 step tutorial on creating automated GIF social sharing cards for your Gatsby blog post using Node.js</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/does-life-get-boring/">Does life as a software developer ever get boring</a></h3><small>December 5, 2020 • ☕️ 2 min read</small></header><p>Motivational speech from an MLH mentor to the December 2020 graduating batch</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/learn-what-not-to-do/">How I learned to manage my time as a new Manager</a></h3><small>December 3, 2020 • ☕️ 5 min read</small></header><p>Learning to manage priorities by deciding what not to do and being productive.</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/do-not-be-a-bottleneck/">Are you a bottleneck in your team?</a></h3><small>November 6, 2020 • ☕️ 5 min read</small></header><p>Unlearn from doing everything yourself and learn to delegate.</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/reality-of-opensource-contributions/">Dear opensource - you are not easy!</a></h3><small>October 16, 2020 • ☕️ 7 min read</small></header><p>My experiences of being both a contributor and a maintainer in the opensource world</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/first-problem-to-solve-as-a-leader/">The first problem to solve as a leader</a></h3><small>October 5, 2020 • ☕️ 5 min read</small></header><p>The problem is people and the solution is people</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/fix-openssl-python-aws/">Managing pythons on MacOS</a></h3><small>September 29, 2020 • ☕️ 2 min read</small></header><p>Handling different python versions on MacOS</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/i-was-interim-cto/">My experience as a new interim CTO</a></h3><small>September 4, 2020 • ☕️ 2 min read</small></header><p>And just like that it happened, I was the interim CTO for two weeks</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/ace-your-outreachy-application/">7 tips to ace your Outreachy application 2021</a></h3><small>August 29, 2020 • ☕️☕️ 10 min read</small></header><p>Lessons and tips on how you can improve your Outreachy internship application</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/are-you-a-new-tech-lead/">Are you a new tech lead?</a></h3><small>August 23, 2020 • ☕️ 3 min read</small></header><p>Are you trying to find tech lead resources and don't know where to start? In this post I talk about what have I found so far</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/to-d3-or-not/">To D3 or not</a></h3><small>August 8, 2020 • ☕️ 2 min read</small></header><p>You may not necessarily need D3.js for visualisations</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/unit-vs-integration-vs-end-to-end-tests/">How to know if you are writing integration tests or end to end tests</a></h3><small>July 22, 2020 • ☕️ 1 min read</small></header><p>Unit tests vs Integration tests vs End to end tests</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/contract-tests-introduction/">Introduction to contract tests</a></h3><small>July 15, 2020 • ☕️ 3 min read</small></header><p>An introduction to contract tests and comparison with integration tests</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/lessons-from-my-younger-self/">What would I advice my younger self</a></h3><small>July 12, 2020 • ☕️ 2 min read</small></header><p>Lessons from my younger self</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/career-advice/">Career advice for freshers</a></h3><small>July 12, 2020 • ☕️ 4 min read</small></header><p>Career advice for freshers</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/okr/">OKRs for a personal career path</a></h3><small>July 11, 2020 • ☕️ 2 min read</small></header><p>Objectives and key results can help you achieve your career goals</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/dart/">Flutter</a></h3><small>July 4, 2020 • ☕️ 1 min read</small></header><p>So I tried Flutter today for the first time</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/two-years-into-leadership/">Two year work anniversary</a></h3><small>July 2, 2020 • ☕️ 1 min read</small></header><p>It's official, I now have 2 years work experience as an engineering leader</p></div></article><article><div><header><h3><a href="https://princiya.com/blog/hello-world/">Hello World</a></h3><small>March 9, 2020 • ☕️ 1 min read</small></header><p>Setting up a Gatsby Blog</p></div></article><article><div><header><h3><a href="https://princiya777.wordpress.com/">From the archives</a></h3></header><p>All old posts can be found under princiya777.wordpress.com.</p></div></article></div></main></div></div>||||I|||| P`s Blog

 Princiya




 Personal blog by Princiya Sequeira.

 I blog about my journey into a technical leadership role.

 I am a skilled programmer with strong communication skills and 12+ years of experience building full-stack web & mobile apps, implementing CI/CD solutions and supporting infrastructures, including 3+ years of experience leading technical teams at scale.

 See all blog categories
 twitter • github • rss




 Featured articles

 How to connect react to redux - a diagrammatic guide
 April 23, 2019


 Here is a diagrammatic guide to better understand how Redux connects to React.

 Web Tracking - what you should know about your privacy online
 April 23, 2018


 This is my attempt to explain how web tracking works in a nutshell.

 WebRTC - a detailed history
 August 06, 2017


 The initial version of the OpenWebRTC implementation was developed internally at Ericsson Research. The latter is maintained by the Google Chrome team.

 Featured categories: leadership (10)mentorship (9)technical (7)

 Is Docker, Kubernetes necessary for frontend development?
 January 7, 2022 • ☕️ 3 min read


 I am an aspiring Front end developer. Currently doing self study on Html, Css, JavaScript and React. I've a question - as a frontend developer do you use Linux, docker, kubernetes?

 technicalq&adevops

 Serverless React
 January 4, 2022 • ☕️ 2 min read


 Serverless React application deployed using Netlify, and an overview of databases to make serverless functions stateful.

 technicaldevops

 How to learn
 December 30, 2021 • ☕️ 3 min read


 Tips on how to learn new things.

 advicementorship

 Thankful and grateful
 November 27, 2021 • ☕️ 3 min read


 There is always something to be thankful for

 update

 How to get the first tech job
 August 24, 2021 • ☕️ 1 min read


 Resources to help you find your first tech job

 resourcesjobs

 Salary negotiation tips for Software Engineers
 May 14, 2021 • ☕️ 6 min read


 3 mistakes to avoid and tips to ace at your next salary negotiation

 mentorshipsalary

 What should you know about salary negotiation?
 May 12, 2021 • ☕️ 5 min read


 How to prepare for your next salary negotiation by knowing what to negotiate

 mentorshipsalary

 MLH Graduation 2021
 April 16, 2021 • ☕️ 1 min read


 Major league hacking graduation speech to the April 2021 graduating batch

 mentorshipadvice

 Finding my focus
 March 21, 2021 • ☕️ 3 min read


 Getting started with the Blogging for Devs Spring Accelerator

 blogging

 How to create automated GIF social sharing cards
 January 8, 2021 • ☕️ 3 min read


 A 3 step tutorial on creating automated GIF social sharing cards for your Gatsby blog post using Node.js

 gatsbytutorial

 Does life as a software developer ever get boring
 December 5, 2020 • ☕️ 2 min read


 Motivational speech from an MLH mentor to the December 2020 graduating batch

 mentorshipadviceq&a

 How I learned to manage my time as a new Manager
 December 3, 2020 • ☕️ 5 min read


 Learning to manage priorities by deciding what not to do and being productive.

 leadershiptransition

 Are you a bottleneck in your team?
 November 6, 2020 • ☕️ 5 min read


 Unlearn from doing everything yourself and learn to delegate.

 leadershiptransition

 Dear opensource - you are not easy!
 October 16, 2020 • ☕️ 7 min read


 My experiences of being both a contributor and a maintainer in the opensource world

 opensourcementorship

 The first problem to solve as a leader
 October 5, 2020 • ☕️ 5 min read


 The problem is people and the solution is people

 leadership

 Managing pythons on MacOS
 September 29, 2020 • ☕️ 2 min read


 Handling different python versions on MacOS

 technicalnotes

 My experience as a new interim CTO
 September 4, 2020 • ☕️ 2 min read


 And just like that it happened, I was the interim CTO for two weeks

 leadership

 7 tips to ace your Outreachy application 2021
 August 29, 2020 • ☕️☕️ 10 min read


 Lessons and tips on how you can improve your Outreachy internship application

 opensourceadvice

 Are you a new tech lead?
 August 23, 2020 • ☕️ 3 min read


 Are you trying to find tech lead resources and don't know where to start? In this post I talk about what have I found so far

 leadership

 To D3 or not
 August 8, 2020 • ☕️ 2 min read


 You may not necessarily need D3.js for visualisations

 d3.jstechnicalvisualisations

 How to know if you are writing integration tests or end to end tests
 July 22, 2020 • ☕️ 1 min read


 Unit tests vs Integration tests vs End to end tests

 testingtechnical

 Introduction to contract tests
 July 15, 2020 • ☕️ 3 min read


 An introduction to contract tests and comparison with integration tests

 testingtechnical

 What would I advice my younger self
 July 12, 2020 • ☕️ 2 min read


 Lessons from my younger self

 leadershipadvicementorship

 Career advice for freshers
 July 12, 2020 • ☕️ 4 min read


 Career advice for freshers

 advicementorship

 OKRs for a personal career path
 July 11, 2020 • ☕️ 2 min read


 Objectives and key results can help you achieve your career goals

 leadershipmentorship

 Flutter
 July 4, 2020 • ☕️ 1 min read


 So I tried Flutter today for the first time

 technical

 Two year work anniversary
 July 2, 2020 • ☕️ 1 min read


 It's official, I now have 2 years work experience as an engineering leader

 leadership

 Hello World
 March 9, 2020 • ☕️ 1 min read


 Setting up a Gatsby Blog

 gatsby

 From the archives


 All old posts can be found under princiya777.wordpress.com.




 Personal blog by Princiya Sequeira.

 I blog about my journey into a technical leadership role.

 I am a skilled programmer with strong communication skills and 12+ years of experience building full-stack web & mobile apps, implementing CI/CD solutions and supporting infrastructures, including 3+ years of experience leading technical teams at scale.

 See all blog categories
 twitter • github • rss




How to connect react to redux - a diagrammatic guide  April 23, 2019    Here is a diagrammatic guide to better understand how Redux connects to React.       Web Tracking - what you should know about your privacy online  April 23, 2018    This is my attempt to explain how web tracking works in a nutshell.       WebRTC - a detailed history  August 06, 2017    The initial version of the OpenWebRTC implementation was developed internally at Ericsson Research. The latter is maintained by the Google Chrome team.         Is Docker, Kubernetes necessary for frontend development?  January 7, 2022 • ☕️ 3 min read  I am an aspiring Front end developer. Currently doing self study on Html, Css, JavaScript and React. I've a question - as a frontend developer do you use Linux, docker, kubernetes?       Serverless React  January 4, 2022 • ☕️ 2 min read  Serverless React application deployed using Netlify, and an overview of databases to make serverless functions stateful.       How to learn  December 30, 2021 • ☕️ 3 min read  Tips on how to learn new things.       Thankful and grateful  November 27, 2021 • ☕️ 3 min read  There is always something to be thankful for       How to get the first tech job  August 24, 2021 • ☕️ 1 min read  Resources to help you find your first tech job       Salary negotiation tips for Software Engineers  May 14, 2021 • ☕️ 6 min read  3 mistakes to avoid and tips to ace at your next salary negotiation       What should you know about salary negotiation?  May 12, 2021 • ☕️ 5 min read  How to prepare for your next salary negotiation by knowing what to negotiate       MLH Graduation 2021  April 16, 2021 • ☕️ 1 min read  Major league hacking graduation speech to the April 2021 graduating batch       Finding my focus  March 21, 2021 • ☕️ 3 min read  Getting started with the Blogging for Devs Spring Accelerator       How to create automated GIF social sharing cards  January 8, 2021 • ☕️ 3 min read    A 3 step tutorial on creating automated GIF social sharing cards for your Gatsby blog post using Node.js       Does life as a software developer ever get boring  December 5, 2020 • ☕️ 2 min read  Motivational speech from an MLH mentor to the December 2020 graduating batch       How I learned to manage my time as a new Manager  December 3, 2020 • ☕️ 5 min read  Learning to manage priorities by deciding what not to do and being productive.       Are you a bottleneck in your team?  November 6, 2020 • ☕️ 5 min read  Unlearn from doing everything yourself and learn to delegate.       Dear opensource - you are not easy!  October 16, 2020 • ☕️ 7 min read  My experiences of being both a contributor and a maintainer in the opensource world       The first problem to solve as a leader  October 5, 2020 • ☕️ 5 min read  The problem is people and the solution is people       Managing pythons on MacOS  September 29, 2020 • ☕️ 2 min read  Handling different python versions on MacOS       My experience as a new interim CTO  September 4, 2020 • ☕️ 2 min read  And just like that it happened, I was the interim CTO for two weeks       7 tips to ace your Outreachy application 2021  August 29, 2020 • ☕️☕️ 10 min read  Lessons and tips on how you can improve your Outreachy internship application       Are you a new tech lead?  August 23, 2020 • ☕️ 3 min read  Are you trying to find tech lead resources and don't know where to start? In this post I talk about what have I found so far       To D3 or not  August 8, 2020 • ☕️ 2 min read  You may not necessarily need D3.js for visualisations       How to know if you are writing integration tests or end to end tests  July 22, 2020 • ☕️ 1 min read  Unit tests vs Integration tests vs End to end tests       Introduction to contract tests  July 15, 2020 • ☕️ 3 min read  An introduction to contract tests and comparison with integration tests       What would I advice my younger self  July 12, 2020 • ☕️ 2 min read  Lessons from my younger self       Career advice for freshers  July 12, 2020 • ☕️ 4 min read  Career advice for freshers       OKRs for a personal career path  July 11, 2020 • ☕️ 2 min read  Objectives and key results can help you achieve your career goals       Flutter  July 4, 2020 • ☕️ 1 min read  So I tried Flutter today for the first time       Two year work anniversary  July 2, 2020 • ☕️ 1 min read  It's official, I now have 2 years work experience as an engineering leader       Hello World  March 9, 2020 • ☕️ 1 min read  Setting up a Gatsby Blog       From the archives   All old posts can be found under princiya777.wordpress.com.
P`s Blog
https://princiya.com/blog/
https://andymatuschak.org/
AndyMatuschak
2019-08-05
None
<div><p>Hi! Iâm a software engineer, designer, and researcher. I work on technologies that expand what people can think and do. </p><div><p>Hi! Iâm a software engineer, designer, and researcher. I work on technologies that expand what people can think and do.</p><p>My methods meander between academia and Silicon Valley: I explore theories by expressing them in real-world systems, which produce insights I use to improve the theories, which I use in turn to improve the systems, and so on.</p><p>Prior to my current work, I helped build iOS at Apple and led <a href="https://early.khanacademy.org/">R&amp;D at Khan Academy</a>.</p><p>Iâm now an independent researcher, supported by a crowdfunded grant from my <a href="https://www.patreon.com/quantumcountry">Patreon</a> community. If you find my work interesting, you can <a href="https://www.patreon.com/quantumcountry">become a member</a> to get patron-only "letters from the lab" and early access to new work.</p></div>||||I|||| Andy
 Matuschak

 Andy Matuschak

 EmailTwitter

 Hi! Iâm a software engineer, designer, and researcher. I work on technologies that expand what people can think and do. moreâ¦

 Hi! Iâm a software engineer, designer, and researcher. I work on technologies that expand what people can think and do.

 My methods meander between academia and Silicon Valley: I explore theories by expressing them in real-world systems, which produce insights I use to improve the theories, which I use in turn to improve the systems, and so on.

 Prior to my current work, I helped build iOS at Apple and led R&D at Khan Academy.

 Iâm now an independent researcher, supported by a crowdfunded grant from my Patreon community. If you find my work interesting, you can become a member to get patron-only "letters from the lab" and early access to new work.

 EmailTwitter
 Want to hear when I publish new work?

 Projects:
 tools for thought

 Projects: tools for thought

 Augmenting cognition and creativity; "how to create magicians?"; "what comes after the book?"; enabling environments.

 What comes after the book?

 1. How can we develop transformative tools for thought?

 Barriers and opportunities, joint with Michael Nielsen

 2. Q uantum Country

 An experimental âmnemonic book,â joint with Michael Nielsen

 3. Why books donât work

 On designing media to reflect how people think and learn

 4. Timeful texts

 Extending the reading experience over time; joint with Michael Nielsen

 More

 More work

 1. Orbit: a platform for the mnemonic medium

 My current project, still under construction

 2. How to write good prompts

 Using spaced repetition to create understanding

 Notes

 Working notes

 1. Whatâs top of mind

 An index of the questions and ideas Iâm exploring

 2. The mnemonic medium

 Design properties, evidence of impact, top questions

 3. Evergreen notes

 Methods for developing ideas over time

 4. Enabling environments, video games, and the Primer

 Incomplete notes on a new kind of growth environment

 5. Programmable attention

 Systems which systematically orchestrate your mind

 6. Taking knowledge work seriously

 Athletes and musicians vs. knowledge workers

 Letters from
 the lab

 Letters from the lab

 Informal essays and project updates. Most are initially written for my patrons, but a subset are later made available to the public.

 Less formal essays (some patron-only)

 1. Three years of crowdfunded research

 Reflections on my membership program

 2. Towards impact through intimacy

 Tentative plans for my 2023 research

 3. Cultivating depth and stillness in research

 Notes on whatÊ¼s worked for me

 4. Lessons from summer 2022 prototype

 Synthesis on new mnemonic medium

 5. Breaking the mnemonic medium out of its box

 Demo/talk of a new floating design

 6. Prospects for consumer silent speech interfaces

 Systems for dictation without audible sound

 7. The joyful surprises of user observation

 Notes from testing a new mnemonic medium

 8. A peritextual mnemonic medium

 Demo/talk of a redesign around reader control

 9. Implicit practice: a sight reading parable

 One way implicit practice fails in knowledge work

 10. Exponentials and forgetting in Quantum Country

 Patterns in preliminary data

 11. Lessons from 2021

 Reflections on creative work, the field, crowdfunding

 12. Tools for thought: science, design, art, craftsmanship?

 Lenses for progress in interface invention

 13. Quantum Countryâs suspiciously flat forgetting curves

 Surprises in analysis of conceptual memory

 14. Doing-centric explanatory mediums

 Board game instruction manualsâ¦ and Figma

 15. Architectures for a more flexible mnemonic medium

 Interface approaches for reader control

 16. Revamping the mnemonic medium for reader control

 Expanding beyond primers' assumptions

 17. Armories for tool-maker/-user collaborations

 Notes on bridging skills and ideas

 18. Finding researchâcontext fit

 A challenge for tool-builders

 19. Crowdfunded research vs. the NSF CAREER grant

 Open-sourcing Orbit; technical collaborations

 20. Too easy to be effortless

 Surprising distributions in Quantum Country

 21. Ratcheting progress in tools for thought

 How to accrete insight as a field?

 22. In search of better questions

 From memory systems to meaning

 23. Reflections on 2020 as an independent researcher

 Funding, culture, limitations, process

 24. Liquid olives and iPhones

 Problem-solving and -finding; The Uncertainty Mindset

 25. Working with authors: entangled skills

 Text-writing needs prompt-writing needs text-writing

 26. The carrying capacity of a regular memory practice

 Tensions with deliberate practice and flow

 27. The galaxy brain problem; speed-running UIs

 Challenges in explaining new mediums

 28. âSkipâ: exponential-backoff deferral mechanismsâ¦

 â¦ and fuzzy inboxes

 29. Thoughts on crowdfunding tools for thought

 Remarks on a milestone in crowdfunding

 30. A nascent art direction for Orbit

 Escaping educational aesthetics

 31. Demonstrating a personal mnemonic medium

 Integrating spaced repetition into prose notes

 32. Bringing ideas into your Orbit

 Extending the desktop for programmable attention

 Projects:
 ed-tech

 Projects: experimental ed-tech

 Work from my time at Khan Academy, where my team worked with teachers to invent novel interactive learning environments.

 Creations from my R&D work at Khan Academy.

 1. Building complex skills online

 Beyond right and wrong: scalable open-ended learning activities

 2. Numbers at play

 Designing digital manipuÂ­latives to reveal numbersâ hidden properties

 3. Playful worlds of creative math

 Reframing early numeracy around adventure, wonder, and creativity

 Blog

 Research Blog

 1. Narrated explorables: three mental models

 When author and learner share a canvas

 2. Students are the subjects of our sentences

 A syntactic trick to emphasize student agency

 3. Looking sideways at the design squiggle

 Reflections on the teamâs design process

 4. Mastery learning and creative tasks

 Mastery learningâs limits for complex skills

 5. Sorting product âbaseball cardsâ

 Bridging behavioral interviews and prototypes

 6. Explanations from âmastersâ of geometry

 The gulf between application and explanation

 7. Safely showing students peersâ comments

 Designing peer interactions to avoid abuse

 8. Games vs. education on unguided instruction

 The limits of teaching without words

 9. Rich tasks which crowdsource more rich tasks

 Using student work to arrange social learning

 10. Surveying the open-ended task landscape

 Notes from literature / market review

 11. Amplifying open-ended questions

 Enabling open-ended online learning activities

 12. One medium, many contexts

 Designing math manipulatives for fluid use

 13. Old and new ways of looking at numbers

 Making abstract properties more visible

 14. A medium for exploring quantity

 Introducing our math manipulative project

 15. Student agency and flow

 Helping students surf their learning edge

 More, misc, meta

 Miscellaneous writing, appearances, and projects scattered across the web.

 Miscellaneous writing, appearances, projects.

 1. Scrying Pen

 A pen which draws its own future using a realtime recurrent neural network

 Podcasts

 Selected podcasts

 1. Econtalk with Russ Roberts

 2019-08-05

 2. Tools & Craft with Devon Zuegel

 2022-06-08

 3. Clearer Thinking with Spencer Greenberg

 2021-07-29

 4. Metamuse with Adam Wiggins and Mark McGranaghan

 2020-09-03

 5. Free Thoughts with Aaron Ross Powell and Trevor Burrus

 2021-01-22

 6. Venture Stories with Erik Torenberg

 2019-10-24

 7. North Star with David Perell

 2020-04-06

 Articles

 Other writing

 1. Exalting data, missing meaning

 The delusions of measurement absent theory

 2. Satisfaction and progress in open-ended work

 Seeking gradient in the fog of novel work

 3. Building habits via smoothly ratcheting targets

 A surprisingly reliable strategy for starting habits

 EmailTwitter
 Want to hear when I publish new work?
Hi! Iâm a software engineer, designer, and researcher. I work on technologies that expand what people can think and do.   Hi! Iâm a software engineer, designer, and researcher. I work on technologies that expand what people can think and do. My methods meander between academia and Silicon Valley: I explore theories by expressing them in real-world systems, which produce insights I use to improve the theories, which I use in turn to improve the systems, and so on. Prior to my current work, I helped build iOS at Apple and led R&amp;D at Khan Academy. Iâm now an independent researcher, supported by a crowdfunded grant from my Patreon community. If you find my work interesting, you can become a member to get patron-only "letters from the lab" and early access to new work.
AndyMatuschak
https://andymatuschak.org/
https://critter.blog/
Mike Crittenden
2024-01-04
Mike Crittenden
<div><div>
<section><h2>Tell me why I’m wrong</h2> <div><p>Feel free to <a href="mailto:mikecrittenden@gmail.com">email me</a> or <a href="https://twitter.com/mcrittenden">tweet me</a> or comment <a href="https://www.reddit.com/r/CritterBlog/">on Reddit</a>.</p>
<p>You’ll get a reply, probably.</p>
</div>
</section>
<section><h2></h2>
<div>
</div>
</section>
<p>
<section><h2>Search for stuff</h2></section> </p>
</div></div>||||I|||| Skip to content

 Mike Crittenden

 Call me Critter

 Menu
 * Subscribe
 * Podcast Expand child menu
 + Apple Podcasts
 + Google Podcasts
 + Spotify
 + PocketCasts
 + Stitcher
 + Breaker
 + RadioPublic
 * Twitter
 * LinkedIn
 * Reddit
 * Archive
 * Search
 January 4, 2024Mike Crittenden

 2023 in review

 January 3, 2024Mike Crittenden

 2023 annual review, kids edition

 January 2, 2024January 2, 2024Mike Crittenden

 If you had to pick one person to spend more time around in 2024…

 December 20, 2023Mike Crittenden

 Handing off control of my screen time

 December 18, 2023Mike Crittenden

 Snakes aren’t poisonous

 December 15, 2023Mike Crittenden

 What DO you want

 December 12, 2023Mike Crittenden

 Back up your performance reviews

 December 7, 2023Mike Crittenden

 Torrenting as analogy for single piece flow

 December 4, 2023Mike Crittenden

 The checklist diet

 December 1, 2023Mike Crittenden

 Senior+ employees aren’t allowed to miss things

 Posts navigation

 Older Posts
 Older posts

 Tell me why I’m wrong

 Feel free to email me or tweet me or comment on Reddit.

 You’ll get a reply, probably.

 Subscribe via email

 Wanna get emailed every time I post something new?

 Email Address:

 Subscribe

 Search for stuff

 Search for:
 Create a website or blog at WordPress.com<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>
Tell me why I’m wrong  Feel free to email me or tweet me or comment on Reddit.
You’ll get a reply, probably.
  
 Search for stuff
Mike Crittenden
https://critter.blog/
https://acoup.blog/
A Collection of Unmitigated Pedantry
2023-12-22
Bret Devereaux
<div><div>
<main>
<article>
<p><a href="https://acoup.blog/2023/12/22/collections-how-many-people-ancient-demography/"></a></p>
<p>One of the first questions anyone asks about past societies is, 'how many people were there?' It is one of those basic bits of information and generally speaking those of us who teach these societies are usually prepared with an estimate to answer the question. But what we often don't talk about is how we come to … <a href="https://acoup.blog/2023/12/22/collections-how-many-people-ancient-demography/">Continue reading </a></p>
</article>
<article>
<p><a href="https://acoup.blog/2023/12/15/collections-shield-walls-and-spacing-hollywood-mobs-and-ancient-tactics/"></a></p>
<p>This week, we're going to take a look at a different aspect of ancient infantry tactics: how heavy infantry shield formations work. While I've framed this around 'shield walls,' not every kind of shielded heavy infantry fought that way and in practice the line between what is a 'shield wall' and what isn't comes down to … <a href="https://acoup.blog/2023/12/15/collections-shield-walls-and-spacing-hollywood-mobs-and-ancient-tactics/">Continue reading </a></p>
</article>
<article>
<p><a href="https://acoup.blog/2023/12/08/fireside-friday-december-8-2023/"></a></p>
<p>Apologies for running a fireside so quickly after the gap week, but with the end of the semester coming as the job market gets busier, I haven't had time to finish the next post on shield walls just yet. That will come out next week for sure though, as it is close to done. I … <a href="https://acoup.blog/2023/12/08/fireside-friday-december-8-2023/">Continue reading </a></p>
</article>
<article>
<p>No blog post this week, folks, as I am both fiercely busy and - by the time this goes up - out of town. We'll be back to our regular schedule next week, with the next big topic I want to tackle being what a shield wall is and how it actually functions on a … <a href="https://acoup.blog/2023/12/01/gap-week-december-1-2023/">Continue reading </a></p>
</article>
<article>
<p><a href="https://acoup.blog/2023/11/24/collections-roman-infantry-tactics-why-the-pilum-and-not-a-spear/"></a></p>
<p>This week's post is intended to answer a question which came up in response to the last post looking at the most common type of Mediterranean spear, which to put it simply is: what is up with the odd Roman heavy infantry kit built around a sword and two javelins (albeit two javelins of an … <a href="https://acoup.blog/2023/11/24/collections-roman-infantry-tactics-why-the-pilum-and-not-a-spear/">Continue reading </a></p>
</article>
<article>
<p><a href="https://acoup.blog/2023/11/17/fireside-friday-november-17-2023/"></a></p>
<p>Hey folks! Fireside this week! I was hoping to have a post on Roman infantry tactics this week, in particular the oddity of the Romans not using spear-and-shield infantry (much), but it isn't ready yet and other things have me quite busy, so fireside it is. Fortunately, we have Ridley Scott to complain about. I … <a href="https://acoup.blog/2023/11/17/fireside-friday-november-17-2023/">Continue reading </a></p>
</article>
<article>
<p><a href="https://acoup.
One of the first questions anyone asks about past societies is, 'how many people were there?' It is one of those basic bits of information and generally speaking those of us who teach these societies are usually prepared with an estimate to answer the question. But what we often don't talk about is how we come to … Continue reading  
   
This week, we're going to take a look at a different aspect of ancient infantry tactics: how heavy infantry shield formations work. While I've framed this around 'shield walls,' not every kind of shielded heavy infantry fought that way and in practice the line between what is a 'shield wall' and what isn't comes down to … Continue reading  
   
Apologies for running a fireside so quickly after the gap week, but with the end of the semester coming as the job market gets busier, I haven't had time to finish the next post on shield walls just yet. That will come out next week for sure though, as it is close to done. I … Continue reading  
No blog post this week, folks, as I am both fiercely busy and - by the time this goes up - out of town. We'll be back to our regular schedule next week, with the next big topic I want to tackle being what a shield wall is and how it actually functions on a … Continue reading  
   
This week's post is intended to answer a question which came up in response to the last post looking at the most common type of Mediterranean spear, which to put it simply is: what is up with the odd Roman heavy infantry kit built around a sword and two javelins (albeit two javelins of an … Continue reading  
   
Hey folks! Fireside this week! I was hoping to have a post on Roman infantry tactics this week, in particular the oddity of the Romans not using spear-and-shield infantry (much), but it isn't ready yet and other things have me quite busy, so fireside it is. Fortunately, we have Ridley Scott to complain about. I … Continue reading
A Collection of Unmitigated Pedantry
https://acoup.blog/
https://rodriguezcommaj.com/blog/
Blog | Jason Rodriguez
2022-11-21
None
<div><div>
	<main>
	<div>
	<p>Follow along <a href="/feed.xml">via RSS</a> to keep up with it all.</p>
 <article>
 <h2><a href="/blog/living-in-data/">Living in Data</a></h2>
 November 21, 2022
 <p></p>
<p>I just finished reading <a href="https://www.jerthorp.com/">Jer Thorp’s Living in Data</a>. Overall, I thought it was an excellent book. It reminded me a bit of one of my all-time favorite nonfiction books, <a href="https://bookshop.org/p/books/life-in-code-a-personal-history-of-technology-ellen-ullman/8480750?ean=9781250181695">Ellen Ullman’s Life in Code</a>. Both are a journey through technology through the lens of personal experience. In Ullman’s case, those experiences are about her different roles in the tech world and how they informed her ideas on life and technology. In Thorp’s case, he takes us through a variety of projects in which he’s participated that all have one thing in common: data.</p>
<p>From river trips and collecting data from remote campsites in Africa to glaciers in Canada and school gymnasiums in redlined St. Louis, each chapter reveals how different people and cultures are dealing with the world’s increasing reliance on data in all of its forms. The usual stuff is in there, like how we’re all constantly tracked by advertisers, but the most interesting stories involve cultures outside of the—or at least my—norm. Cultures like the Māori in New Zealand think differently about data and data ownership. Their concept of tāonga—translated as “treasure”—applies not only to physical possessions, but resources like ideas and cultural traditions, too. It’s a deep-rooted cultural concept that impacts how modern day Māori think about data sovereignty and, more importantly, how people outside of that culture can legally interact with Māori data. Applied to the western world, it could (and I’d argue <em>should</em>) impact how companies and individuals think about data.</p>
<p>The book is subtitled, “A citizen’s guide to a better information future.” Although it was thoroughly enjoyable, and got me excited about different data-driven futures, it’s definitely less of a “guide” than a memoir of Thorp’s work. There wasn’t a lot of practical guidance for living a day-to-day life in a data-driven world, but perhaps that’s because everything’s so messy right now and no one knows the right way to approach data across such disparate cultures. What works for the Māori might now work for someone from Idaho. While we can take loose lessons from how different people think about data around the world, it’s harder to distill those lessons into actionable guidance that people can apply today. Which browsers or services should I use? How do I get a company to delete data about me? How can I protect my data in the future? Those are hard questions without a single, one-size-fits-all answer.</p>
<p>Still, they are important questions to consider and Thorp did a great job getting me excited to think about them more deeply.</p>
 </article>
 <article>
 <h2><a href="/blog/adios-twitter/">Adios, Twitter</a></h2>
 November 20, 2022
 <p></p>
<p>Twitter played an important role in my life and career over the last decade or so. But not anymore. I refuse to participate in the hellscape that Elon Musk is creating.</p>
<p>I joined Twitter in June of 2011. In the years since, Twitter has introduced me to people I now call friends, helped me to grow my career and side projects, and connect with a number of communities in meaningful ways. Although I’ve been mostly inactive on the platform for the last year or two, I still checked it regularly to keep up on what folks were doing in those communities.</p>
<p>Since Musk started eviscerating the platform (and all of the unfortunate souls who worked on the product), damn near everyone I still care to follow has set up on Mastodon, dusted off their RSS feeds, or started up newsletters. Over the last week, my feed has slowed to a crawl of mostly, “See ya, Twitter… follow me here.” posts and more sad, heartbreaking news. Mastodon, on the other hand, has exploded. While my feed there is a lot of Twitter-related analysis, there are people sharing life updates, thoughtful posts, and helpful tips, too.</p>
<p>I used to get value out of Twitter. As of this past week or so, there’s been virtually zero value derived from the platform other than the pure schadenfreude of watching Musk waste billions of dollars and destroy a tool that’s been increasingly harmful to the world over the last few years. It’d be more satisfying if it didn’t affect so many people’s lives.</p>
<p>I don’t want to use Twitter. I don’t want to syndicate to Twitter. I don’t want to contribute to Elon Musk’s monthly active user count. So, I downloaded all my data, updated my profile to make it clear where to find me, and deleted Tweetbot from my devices. I’m not deleting my account, since that would open up my handle for someone else and I don’t want to deal with any repercussions from that. But, I’ll no longer be active on the platform.</p>
<p>I encourage everyone to make plans for their exodus from Twitter if possible (I know it’s not for a lot of people). As always, you can follow <a href="https://rodriguezcommaj.com/feed.xml">via RSS</a>, my occasional <a href="https://rodriguezcommaj.com/newsletter/">email newsletter</a>, or <a href="https://mastodon.social/@rodriguezcommaj">on Mastodon</a>.</p>
 </article>
 <article>
 <h2><a href="/blog/getting-smaller/">Getting Smaller</a></h2>
 November 16, 2022
 <p>As someone who’s worked in tech for a long time, most days find me complaining about the industry as opposed to praising any supposed “advances” it makes. It’s honestly hard not to these days.</p>
<p>But sometimes, you find yourself pleasantly surprised by tech progress. Yesterday was one of those times.</p>
<p>A number of years ago, I started experiencing some symptoms that led to a diagnosis for a mild heart condition. Nothing life-threatening or untreatable. Beyond taking medication, the condition’s major impact on my life has been forcing me to see a cardiologist a few times a year, getting occasional EKGs, and wearing a heart monitor from time-to-time.</p>
<p>The first heart monitor—or Holter monitor, to be more accurate—I had to wear was a bulky collection of sticky electrodes, wires, and a big ol’ box (the recorder) to collect the data, which I wore like the world’s worst necklace. I’d put it underneath my shirt but who am I kidding? You couldn’t hide that thing.</p>
<p></p>
<p>The next one I remember was similar, but the recorder was slightly smaller and fit almost comfortably in my pants pocket. Eventually, the technology progressed enough that the electrodes didn’t need to be physically wired to the recorder and the recorder was replaced with a cheap, generic Android phone which fit much better in my pocket. Still annoying to carry around two phones, but an improvement nonetheless…</p>
<p></p>
<p>At some point in the last few years, they were able to ditch the phone entirely and combine the recorder and electrodes into an assembly that looked like my ribs flying a kite on my chest. Nothing external to carry around and—apart from the irritation from shaving parts of my torso—relatively comfortable.</p>
<p></p>
<p>As I’m writing this, I’m wearing what will hopefully be my last Holter monitor. It’ll see how my heart’s doing after my doctor decided to try taking my off my medication. When I got it put on yesterday, I expected at least two sections and a wire, similar to the last one. What I got instead was what looks like a Tamagochi stuck to my chest. Not quite as fun as a Tamagochi, and with a lot more razor burn, but I marveled at how small the monitor is now. It was a far cry from the box, wires, and stickers that I had to use less than a decade ago.</p>
<p></p>
<p>So far, I’ve barely noticed it. It’s lightweight, compact, and mostly hidden underneath a sweatshirt (thanks, chilly Michigan weather). Unless I feel any symptoms, I don’t even have to think about it. And, if I do feel anything, all I need to do is press it to log an event. Early next week, I’ll peel it off, stick it in a box with a prepaid label, and send it on its way. Next month, my doctor and I will see what it says and figure out next steps. Contrast that to the first few monitors, which required me to drive 45 minutes into the office and have a tech remove it and send it to the lab.</p>
<p>It’s easy to get lost in the drudgery of the tech world. Sure, iPhones get better cameras every year, but the tech world has shown its capacity to wreak havoc on the world and its inhabitants in countless ways over the past few years. But this week has reminded me that tech can still improve people’s experiences and lives in small but still important ways.</p>
 </article>
	</div>
	</main>
	</div>||||I|||| Jason Rodriguez
 About • Blog • Blogroll • Concert List • Newsletter • Reading List • Work

 Blog

 Follow along via RSS to keep up with it all.



 Living in Data

 November 21, 2022

 I just finished reading Jer Thorp’s Living in Data. Overall, I thought it was an excellent book. It reminded me a bit of one of my all-time favorite nonfiction books, Ellen Ullman’s Life in Code. Both are a journey through technology through the lens of personal experience. In Ullman’s case, those experiences are about her different roles in the tech world and how they informed her ideas on life and technology. In Thorp’s case, he takes us through a variety of projects in which he’s participated that all have one thing in common: data.

 From river trips and collecting data from remote campsites in Africa to glaciers in Canada and school gymnasiums in redlined St. Louis, each chapter reveals how different people and cultures are dealing with the world’s increasing reliance on data in all of its forms. The usual stuff is in there, like how we’re all constantly tracked by advertisers, but the most interesting stories involve cultures outside of the—or at least my—norm. Cultures like the Māori in New Zealand think differently about data and data ownership. Their concept of tāonga—translated as “treasure”—applies not only to physical possessions, but resources like ideas and cultural traditions, too. It’s a deep-rooted cultural concept that impacts how modern day Māori think about data sovereignty and, more importantly, how people outside of that culture can legally interact with Māori data. Applied to the western world, it could (and I’d argue should) impact how companies and individuals think about data.

 The book is subtitled, “A citizen’s guide to a better information future.” Although it was thoroughly enjoyable, and got me excited about different data-driven futures, it’s definitely less of a “guide” than a memoir of Thorp’s work. There wasn’t a lot of practical guidance for living a day-to-day life in a data-driven world, but perhaps that’s because everything’s so messy right now and no one knows the right way to approach data across such disparate cultures. What works for the Māori might now work for someone from Idaho. While we can take loose lessons from how different people think about data around the world, it’s harder to distill those lessons into actionable guidance that people can apply today. Which browsers or services should I use? How do I get a company to delete data about me? How can I protect my data in the future? Those are hard questions without a single, one-size-fits-all answer.

 Still, they are important questions to consider and Thorp did a great job getting me excited to think about them more deeply.

 Adios, Twitter

 November 20, 2022

 Twitter played an important role in my life and career over the last decade or so. But not anymore. I refuse to participate in the hellscape that Elon Musk is creating.

 I joined Twitter in June of 2011. In the years since, Twitter has introduced me to people I now call friends, helped me to grow my career and side projects, and connect with a number of communities in meaningful ways. Although I’ve been mostly inactive on the platform for the last year or two, I still checked it regularly to keep up on what folks were doing in those communities.

 Since Musk started eviscerating the platform (and all of the unfortunate souls who worked on the product), damn near everyone I still care to follow has set up on Mastodon, dusted off their RSS feeds, or started up newsletters. Over the last week, my feed has slowed to a crawl of mostly, “See ya, Twitter… follow me here.” posts and more sad, heartbreaking news. Mastodon, on the other hand, has exploded. While my feed there is a lot of Twitter-related analysis, there are people sharing life updates, thoughtful posts, and helpful tips, too.

 I used to get value out of Twitter. As of this past week or so, there’s been virtually zero value derived from the platform other than the pure schadenfreude of watching Musk waste billions of dollars and destroy a tool that’s been increasingly harmful to the world over the last few years. It’d be more satisfying if it didn’t affect so many people’s lives.

 I don’t want to use Twitter. I don’t want to syndicate to Twitter. I don’t want to contribute to Elon Musk’s monthly active user count. So, I downloaded all my data, updated my profile to make it clear where to find me, and deleted Tweetbot from my devices. I’m not deleting my account, since that would open up my handle for someone else and I don’t want to deal with any repercussions from that. But, I’ll no longer be active on the platform.

 I encourage everyone to make plans for their exodus from Twitter if possible (I know it’s not for a lot of people). As always, you can follow via RSS, my occasional email newsletter, or on Mastodon.

 Getting Smaller

 November 16, 2022

 As someone who’s worked in tech for a long time, most days find me complaining about the industry as opposed to praising any supposed “advances” it makes. It’s honestly hard not to these days.

 But sometimes, you find yourself pleasantly surprised by tech progress. Yesterday was one of those times.

 A number of years ago, I started experiencing some symptoms that led to a diagnosis for a mild heart condition. Nothing life-threatening or untreatable. Beyond taking medication, the condition’s major impact on my life has been forcing me to see a cardiologist a few times a year, getting occasional EKGs, and wearing a heart monitor from time-to-time.

 The first heart monitor—or Holter monitor, to be more accurate—I had to wear was a bulky collection of sticky electrodes, wires, and a big ol’ box (the recorder) to collect the data, which I wore like the world’s worst necklace. I’d put it underneath my shirt but who am I kidding? You couldn’t hide that thing.

 The next one I remember was similar, but the recorder was slightly smaller and fit almost comfortably in my pants pocket. Eventually, the technology progressed enough that the electrodes didn’t need to be physically wired to the recorder and the recorder was replaced with a cheap, generic Android phone which fit much better in my pocket. Still annoying to carry around two phones, but an improvement nonetheless…

 At some point in the last few years, they were able to ditch the phone entirely and combine the recorder and electrodes into an assembly that looked like my ribs flying a kite on my chest. Nothing external to carry around and—apart from the irritation from shaving parts of my torso—relatively comfortable.

 As I’m writing this, I’m wearing what will hopefully be my last Holter monitor. It’ll see how my heart’s doing after my doctor decided to try taking my off my medication. When I got it put on yesterday, I expected at least two sections and a wire, similar to the last one. What I got instead was what looks like a Tamagochi stuck to my chest. Not quite as fun as a Tamagochi, and with a lot more razor burn, but I marveled at how small the monitor is now. It was a far cry from the box, wires, and stickers that I had to use less than a decade ago.

 So far, I’ve barely noticed it. It’s lightweight, compact, and mostly hidden underneath a sweatshirt (thanks, chilly Michigan weather). Unless I feel any symptoms, I don’t even have to think about it. And, if I do feel anything, all I need to do is press it to log an event. Early next week, I’ll peel it off, stick it in a box with a prepaid label, and send it on its way. Next month, my doctor and I will see what it says and figure out next steps. Contrast that to the first few monitors, which required me to drive 45 minutes into the office and have a tech remove it and send it to the lab.

 It’s easy to get lost in the drudgery of the tech world. Sure, iPhones get better cameras every year, but the tech world has shown its capacity to wreak havoc on the world and its inhabitants in countless ways over the past few years. But this week has reminded me that tech can still improve people’s experiences and lives in small but still important ways.

 Travel Through Time

 Older → Page: 1 of 22

 GitHub • Mastodon • LinkedIn

 © 2012 - 2023 Jason Rodriguez. All rights reserved. Made in Michigan. Opinions are mine and not necessarily shared by the folks I work for and with.

 Say hello or subscribe via RSS.

Follow along via RSS to keep up with it all.
 
  Living in Data 
 November 21, 2022
  
I just finished reading Jer Thorp’s Living in Data. Overall, I thought it was an excellent book. It reminded me a bit of one of my all-time favorite nonfiction books, Ellen Ullman’s Life in Code. Both are a journey through technology through the lens of personal experience. In Ullman’s case, those experiences are about her different roles in the tech world and how they informed her ideas on life and technology. In Thorp’s case, he takes us through a variety of projects in which he’s participated that all have one thing in common: data.
From river trips and collecting data from remote campsites in Africa to glaciers in Canada and school gymnasiums in redlined St. Louis, each chapter reveals how different people and cultures are dealing with the world’s increasing reliance on data in all of its forms. The usual stuff is in there, like how we’re all constantly tracked by advertisers, but the most interesting stories involve cultures outside of the—or at least my—norm. Cultures like the Māori in New Zealand think differently about data and data ownership. Their concept of tāonga—translated as “treasure”—applies not only to physical possessions, but resources like ideas and cultural traditions, too. It’s a deep-rooted cultural concept that impacts how modern day Māori think about data sovereignty and, more importantly, how people outside of that culture can legally interact with Māori data. Applied to the western world, it could (and I’d argue should) impact how companies and individuals think about data.
The book is subtitled, “A citizen’s guide to a better information future.” Although it was thoroughly enjoyable, and got me excited about different data-driven futures, it’s definitely less of a “guide” than a memoir of Thorp’s work. There wasn’t a lot of practical guidance for living a day-to-day life in a data-driven world, but perhaps that’s because everything’s so messy right now and no one knows the right way to approach data across such disparate cultures. What works for the Māori might now work for someone from Idaho. While we can take loose lessons from how different people think about data around the world, it’s harder to distill those lessons into actionable guidance that people can apply today. Which browsers or services should I use? How do I get a company to delete data about me? How can I protect my data in the future? Those are hard questions without a single, one-size-fits-all answer.
Still, they are important questions to consider and Thorp did a great job getting me excited to think about them more deeply.
 
 
  Adios, Twitter 
 November 20, 2022
  
Twitter played an important role in my life and career over the last decade or so. But not anymore. I refuse to participate in the hellscape that Elon Musk is creating.
I joined Twitter in June of 2011. In the years since, Twitter has introduced me to people I now call friends, helped me to grow my career and side projects, and connect with a number of communities in meaningful ways. Although I’ve been mostly inactive on the platform for the last year or two, I still checked it regularly to keep up on what folks were doing in those communities.
Since Musk started eviscerating the platform (and all of the unfortunate souls who worked on the product), damn near everyone I still care to follow has set up on Mastodon, dusted off their RSS feeds, or started up newsletters. Over the last week, my feed has slowed to a crawl of mostly, “See ya, Twitter… follow me here.” posts and more sad, heartbreaking news. Mastodon, on the other hand, has exploded. While my feed there is a lot of Twitter-related analysis, there are people sharing life updates, thoughtful posts, and helpful tips, too.
I used to get value out of Twitter. As of this past week or so, there’s been virtually zero value derived from the platform other than the pure schadenfreude of watching Musk waste billions of dollars and destroy a tool that’s been increasingly harmful to the world over the last few years. It’d be more satisfying if it didn’t affect so many people’s lives.
I don’t want to use Twitter. I don’t want to syndicate to Twitter. I don’t want to contribute to Elon Musk’s monthly active user count. So, I downloaded all my data, updated my profile to make it clear where to find me, and deleted Tweetbot from my devices. I’m not deleting my account, since that would open up my handle for someone else and I don’t want to deal with any repercussions from that. But, I’ll no longer be active on the platform.
I encourage everyone to make plans for their exodus from Twitter if possible (I know it’s not for a lot of people). As always, you can follow via RSS, my occasional email newsletter, or on Mastodon.
 
 
  Getting Smaller 
 November 16, 2022
 As someone who’s worked in tech for a long time, most days find me complaining about the industry as opposed to praising any supposed “advances” it makes. It’s honestly hard not to these days.
But sometimes, you find yourself pleasantly surprised by tech progress. Yesterday was one of those times.
A number of years ago, I started experiencing some symptoms that led to a diagnosis for a mild heart condition. Nothing life-threatening or untreatable. Beyond taking medication, the condition’s major impact on my life has been forcing me to see a cardiologist a few times a year, getting occasional EKGs, and wearing a heart monitor from time-to-time.
The first heart monitor—or Holter monitor, to be more accurate—I had to wear was a bulky collection of sticky electrodes, wires, and a big ol’ box (the recorder) to collect the data, which I wore like the world’s worst necklace. I’d put it underneath my shirt but who am I kidding? You couldn’t hide that thing.
 
The next one I remember was similar, but the recorder was slightly smaller and fit almost comfortably in my pants pocket. Eventually, the technology progressed enough that the electrodes didn’t need to be physically wired to the recorder and the recorder was replaced with a cheap, generic Android phone which fit much better in my pocket. Still annoying to carry around two phones, but an improvement nonetheless…
 
At some point in the last few years, they were able to ditch the phone entirely and combine the recorder and electrodes into an assembly that looked like my ribs flying a kite on my chest. Nothing external to carry around and—apart from the irritation from shaving parts of my torso—relatively comfortable.
 
As I’m writing this, I’m wearing what will hopefully be my last Holter monitor. It’ll see how my heart’s doing after my doctor decided to try taking my off my medication. When I got it put on yesterday, I expected at least two sections and a wire, similar to the last one. What I got instead was what looks like a Tamagochi stuck to my chest. Not quite as fun as a Tamagochi, and with a lot more razor burn, but I marveled at how small the monitor is now. It was a far cry from the box, wires, and stickers that I had to use less than a decade ago.
 
So far, I’ve barely noticed it. It’s lightweight, compact, and mostly hidden underneath a sweatshirt (thanks, chilly Michigan weather). Unless I feel any symptoms, I don’t even have to think about it. And, if I do feel anything, all I need to do is press it to log an event. Early next week, I’ll peel it off, stick it in a box with a prepaid label, and send it on its way. Next month, my doctor and I will see what it says and figure out next steps. Contrast that to the first few monitors, which required me to drive 45 minutes into the office and have a tech remove it and send it to the lab.
It’s easy to get lost in the drudgery of the tech world. Sure, iPhones get better cameras every year, but the tech world has shown its capacity to wreak havoc on the world and its inhabitants in countless ways over the past few years. But this week has reminded me that tech can still improve people’s experiences and lives in small but still important ways.
Blog | Jason Rodriguez
https://rodriguezcommaj.com/blog/
https://pedestrianobservations.com/
Pedestrian Observations
2023-01-30
Alon Levy
<div><section>
	<article>
	<header>
	<time>2023-01-30</time>
	<h2>
	<a href="https://pedestrianobservations.com/2023/01/30/fdp-and-vice-signaling/">
	FDP and Vice Signaling	</a>
	</h2>
	</header>
	<div>
	
<p>Finance Minister Christian Lindner (FDP) <a href="https://twitter.com/c_lindner/status/1619992524952866817">just tweeted</a> that more investment in roads is good – because if traffic flows more smoothly then there will be less greenhouse gas emissions. Reaction was not positive, and as of when I’m writing, 16 hours later, it is mildly ratioed. People understand that this is wrong. Lindner himself probably gets this too. Understanding what’s going on here requires talking about bullshit in the philosophical sense of Harry Frankfurt, and about something that I don’t have a better name for than vice signaling.</p>
<p><strong>Is it true?</strong></p>
<p>Absolutely not. It’s standard in transport studies that the construction of more highways in high-demand areas induces more traffic, as people take advantage of the greater convenience of driving. Drivers drive to new destinations that they forwent or chose to take public transport to, and new developments are built in areas opened by new highway development.</p>
<p>There may be exceptions to this in declining areas. The United States loves building new grade-separated interchanges in declining regions. This doesn’t generate new demand, because traffic is already uncongested, and the purpose of roadbuilding there is a political statement more than transport policy. But that’s not Germany. The roads under discussion here are in growth regions: there’s a plan to widen the beltway around Munich, A99, to 10 lanes, and the federal and Berlin FDP have both badgered Berlin to build a further stage of A100 parallel to the Ringbahn, which the city wants not to under the influence of the Green Party. Both motorway projects are likely to lead to adverse mode shift if built, and Lindner knows this.</p>
<p>There’s a developmental argument that induced demand is actually good. Matt Yglesias has made it before, saying that if road building induces more traffic then it means people get to take more trips and are better off. Many roadbuilders have made that very argument, and others were aware of it; Robert Moses, for example, was perfectly aware that his parkways and bridges were inducing more car traffic, and was fine with it, because he thought more driving was good. But that’s not what Lindner is saying: Lindner is saying that building new motorways and keeping them without a speed limit reduces greenhouse gas emissions, which is just bullshit.</p>
<p><strong>Bullshit</strong></p>
<p>The term “bullshit” has a precise meaning in analytic philosophy, <a href="http://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on_bullshit.pdf">due to Harry Frankfurt</a>. It comprises a type of deception about the speaker’s mindset, rather than about the facts, unlike an ordinary lie. A politician who denies a scandal they are involved with is lying: their goal is to get you to believe that they are innocent of this scandal. A politician who, having been caught in said scandal, launches a series of schlock patriotic speeches is bullshitting: their goal is to get you to think they are fundamentally aligned with your values. From Frankfurt’s original essay, we have,</p>
<blockquote>
<p>Telling a lie is an act with a sharp focus. It is designed to insert a particular falsehood at a specific point in a set or system of beliefs, in order to avoid the consequences of having that point occupied by the truth. This requires a degree of craftsmanship, in which the teller of the lie submits to objective constraints imposed by what he takes to be the truth. The liar is inescapably concerned with truth-values. In order to invent a lie at all, he must think he knows what is true. And in order to invent an effective lie, he must design his falsehood under the guidance of that truth. On the other hand, a person who undertakes to bullshit his way through has much more freedom. His focus is panoramic rather than particular. He does not limit himself to inserting a certain falsehood at a specific point, and thus he is not constrained by the truths surrounding that point or intersecting it. He is prepared to fake the context as well, so far as need requires. This freedom from the constraints to which the liar must submit does not necessarily mean, of course, that his task is easier than the task of the liar. But the mode of creativity upon which it relies is less analytical and less deliberative than that which is mobilized in lying. It is more expansive and independent, with mare spacious opportunities for improvisation, color, and imaginative play. This is less a matter of craft than of art. Hence the familiar notion of the “bullshit artist.”</p>
</blockquote>
<p>The statement “widening roads reduces CO2 emissions” is this kind of bullshit. It is not quite a lie: it is false, but Lindner is not especially concerned with whether it is true or false. His goal is not to persuade people that building another section of A100 and widening A99 is good for climate; nobody who cares about climate change thinks that. Rather, his goal is to position himself as the sort of person who doesn’t listen to climate advocates and will just push for road widenings. The deception is part of the positioning: if he’d said that he understands the Greens’ argument against road investment but roads are important for economic development, he’d come off as too reasonable, which is not his intention.</p>
<p>Sounding deliberately unreasonable is the domain of populist politicians, and Frankfurt himself and many of his followers have noticed how political bullshit is on the rise as populism grows more normalized. Nigel Farage, for example, <a href="https://uk.news.yahoo.com/nigel-farage-dismisses-apos-clever-134131194.html">bullshitted that smoking isn’t bad for your health</a>. And FDP is a populist party, despite its liberal origins and relatively moderate political positioning; it swung from deficit scold at the start of the current government to tax scold precisely as inflation rose last year, the opposite of what one should expect of a Washington Consensus-following economically orthodox party.</p>
<p><strong>Vice signaling</strong></p>
<p>There’s a pseudo-academic term going around the web, <em>virtue signaling</em>. The idea is that individuals and organizations engage in actions to signal that they’re better people than they really are; companies hire consultants on diversity, equity, and inclusion (DEI) without ever doing anything about their glass ceiling and harassment problems.</p>
<p>But it may be more fruitful to discuss its opposite – that is, vice signaling. This is when people take actions to portray themselves as terrible people, for any number of reasons:</p>
<ul>
<li>Loyalty: criminal gangs are deliberately threatening and often require that prospective members commit murder (this is a requirement to become a made man in the Italian-American mafia), because this forces new members to have crossed both a moral and a legal event horizon from which they can’t come back; populist political movements don’t require crimes, but do require ridiculous beliefs</li>
<li>Novelty: this is what in the online language of the early 2010s was called the Slate Pitch – a take that aims to be novel by saying something really out there, often by writers who can’t separate themselves from the rest of the pack by any more productive means</li>
<li>Love of power: some people lie to you, with your full knowledge that they’re lying, just to flex that they can get away with it</li>
</ul>
<p>Lindner loves this kind of vice signaling, I think out of novelty more than anything. FDP could be a party of YIMBYism, fiscal conservatism, and digital governance; younger members of the party who identify with neoliberalism wish that it were that party. The problem is that the difference between such a party and SPD is not large; Scholz ran on building more housing Germany-wide, and there’s a fair amount of consensus in favor of this in the party’s wings. SPD’s worst attributes so far are its <a href="https://www.worldpoliticsreview.com/in-buying-russian-gas-germany-were-played-for-fools/">officious leadership</a> anchored in the <a href="https://foreignpolicy.com/2023/01/29/the-true-power-center-of-germany-isnt-berlin/?tpcc=recirc_latest062921">Lower Saxony clique</a> and consequently its sluggish governance and refusal to do more to support Ukraine – but FDP has the exact same problems, <a href="https://kyivindependent.com/uncategorized/ambassador-says-german-minister-was-against-helping-ukraine-thought-it-would-fall-within-hours">Lindner having told Ukraine when it asked for aid as the war started that there was no point</a> since they’d fall in hours either way.</p>
<p>So to distinguish themselves from everyone else, FDP engages in vice signaling about climate and transport. They’re not trying to convince anyone that their policies are good for climate change. Rather, they’re doing the exact opposite: they’re trying to convince center-right voters that they’re an internal opposition within a coalition that is engaging in modal shift in federal funding priorities, and that they are explicitly <em>against</em> any climate action, because cars are good and only annoying hippies prefer trains.</p>
	</div>
	</article>
	<article>
	<header>
	<time>2023-01-28</time>
	<h2>
	<a href="https://pedestrianobservations.com/2023/01/28/quick-note-high-speed-2-and-euston/">
	Quick Note: High Speed 2 and Euston	</a>
	</h2>
	</header>
	<div>
	
<p>There was reporting in the Sun, <a href="https://www.bbc.com/news/uk-64421566">since officially denied</a>, that Britain is planning to cut Euston from its high-speed rail project and run trains only as far a Old Oak Common, a future development site west of Central London. I assume given the source and lack of any other confirmation that the plans are to run to Euston as planned. But what if the story is not completely fake news, and there are plans to cut on construction at Euston? I can see a cut being positive value engineering, using space at the station more efficiently.</p>
<p><strong>What’s the issue with High Speed 2?</strong></p>
<p>High Speed 2 is an extremely expensive line. Among proper intercity high-speed rail lines (as opposed to suburban lines running at medium speed), it is the most expensive in <a href="https://transitcosts.com/high-speed-rail/">our database</a> per kilometer. The projected cost as of 2019-20 is about the same as that of all lines built to date in Germany and France combined; Germany has about 1,000 km of newly-built high-speed lines and France 2,500, whereas HS2 is planned to be 530 km.</p>
<p>The high costs are related to some massive scope creep, including tunnels in relatively flat terrain through the Chilterns, dug essentially because the area has rich NIMBYs and the British state decided not to fight them. Those are already in advanced enough construction that I don’t think descoping them and building the line at-grade with compulsory purchase of land is viable. However, some of the scope is new stations, which British defenders of the system insist are necessary. Birmingham is to get an entirely new station at Curzon Street, and London Euston is to get a substantial increase in size, with additional tracks and approaches. This is said to be necessary for capacity reasons.</p>
<p><strong>Are new stations necessary for capacity?</strong></p>
<p>No. Euston today has 16 platform tracks; it had 18 before HS2 construction started. The S-Bahn-quality Watford DC line can use two; the remaining slow services at the station amount to around 10-12 trains per hour, which S-Bahn-quality terminals like Saint-Lazare on the RER E and Catalunya on the Barcelona Rodalies network can comfortably turn on four tracks; those two comparisons turn 16 and 24 trains per hour on four tracks, respectively. The services out of Euston branch more than the RER and Rodalies, but this is mostly a mix of stopping patterns, largely on the same legacy line.</p>
<p>Then there’s HS2 itself. The line is expected to get very high ridership, justifiably: all cities along the line are larger than their comparison cases on the LGV Sud-Est, often substantially, and the projection is that very high capacity is required, on the order of 16 trains per hour. This stretches high-speed rail to its limit: the Shinkansen, which mixes local and express trains on double track, peaks around 14-15 trains per hour, and the complexly branched TGV around 12. HS2 expects to do better perhaps through better signals but also through having a simpler stopping pattern on its most congested section, between London and the bifurcation at Birmingham Interchange, on which trains are to run nonstop.</p>
<p>However, 16 trains per hour can still turn on about six platform tracks. This is not easy: <a href="https://www.todayonline.com/world/asia/cleaning-brigade-helps-japans-bullet-trains-run-time">the Tohoku Shinkansen turns 14 on four tracks</a>, but this is a limit case, famous not just in rail media but also in business media as successful optimization of infrastructure. Nonetheless, given how constrained the site is, it’s useful to learn from the best and not the average. If it’s possible to descope the plans to add new tracks to Euston, this should be done; present plans for Euston cost billions.</p>
<p><strong>Is this happening?</strong></p>
<p>Maybe. Britain is aware of the situation at Tokyo Station, although it seems more interested in looking for reasons not to learn than in learning. Perhaps very high costs are leading to a reevaluation, in which Euston can be made smaller than in current plans and trains can turn more efficiently.</p>
<p>But again, the ultimate source on this said nothing of this sort, and is unreliable. So who knows?</p>
	</div>
	</article>
	<article>
	<header>
	<time>2023-01-26</time>
	<h2>
	<a href="https://pedestrianobservations.com/2023/01/26/free-public-transport-fare-integration-and-capacity/">
	Free Public Transport, Fare Integration, and Capacity	</a>
	</h2>
	</header>
	<div>
	
<p>There’s an ongoing debate about free public transport that I’m going to get into later, but, for now, I want to zoom in on one aspect of the <a href="https://pedestrianobservations.com/2022/08/31/the-nine-euro-ticket/">9€ ticket</a>, and how it impacted public transport capacity in Germany. A commenter on the Neoliberal Reddit group claimed that during the three months of nearly free public transport fares, there was a capacity crunch due to overuse. But in fact, the impact was not actually significant on urban rail, only on regional trains, in a way that underscores the importance of fare integration more than anything.</p>
<p><strong>What was the 9€ ticket?</strong></p>
<p>Last year, in the wake of the Russian invasion of Ukraine, fuel prices shot up everywhere. This created populist pressure to alleviate the price of fuel through temporary tax cuts, which further exacerbated last year’s high inflation. The center-right element within the German coalition, FDP, moved away from its traditional position as deficit scold and demanded a cut in the fuel tax; as a compromise, the Greens agreed to it on condition that during the three months of reduced fuel tax, June through August, public transport fares would be cut as well. Thus the 9€ monthly was born.</p>
<p>The 9€ ticket applied throughout Germany. The key feature wasn’t just the deep discount but also the fact that on one ticket, people could travel all over Germany; normally, my Berlin monthly doesn’t let me ride the local trains in Leipzig or Munich. This stimulated massive domestic tourism, since people could travel between cities on slow regional trains for free and then also travel around their destination city for free as well.</p>
<p><strong>What now?</strong></p>
<p>The 9€ ticket clearly raised public transport ridership in the three months it was in effect. This led to demands to make it permanent, running up against the problem that money is scarce and in Germany ticket fares generate a significant proportion of public transport revenue, 7.363 billion € out of 14.248 billion € in expenses (<a href="https://www.vdv.de/vdv-statistik-2019.pdfx">source</a>, p. 36).</p>
<p>One partial move in that direction is a 29€ monthly valid only within Berlin, not in the suburbs (zone C of the S-Bahn) or outside the system; unlike the 9€ ticket, which was well-advertised all over national and local media and was available at every ticketing machine, the 29€ monthly is only available via annual subscription, which requires a permanent address in the city, and the regular machines only sell the usual 86€ monthly and don’t even let you know that a cheaper option exists. The subscription is also not available on a rolling basis – one must do it before the start of the month, which is not advertised, and <a href="https://www.cat-bus.com/">Ant6n</a>‘s family was caught unaware one month.</p>
<p>Negotiations for a nationwide 49€ ticket are underway, proceeding at the pace of a German train, or perhaps that of German arms deliveries to Ukraine. This was supposed to start at the beginning of 2023, then in April, and <a href="https://www.ndr.de/ratgeber/verbraucher/49-Euro-Ticket-Wann-kommt-es-und-wo-gilt-es,deutschlandticket104.html">now it’s expected to debut in May</a>. I’m assuming it will eventually happen – German trains get you there eventually, if hours late occasionally.</p>
<p><strong>What’s the impact on capacity?</strong></p>
<p>The U- and S-Bahn systems didn’t at all get overcrowded. They got a bit more crowded than usual, but nothing especially bad, since the sort of trips induced by zero marginal cost are off-peak. Rush hour commuters are not usually price-sensitive: whenever one’s alternative to the train is a car, the difference between a 9€ monthly and an 86€ one is a fraction of the difference between either ticket and the cost of owning and using a car, and at rush hour, cars are limited by congestion as well. Off-peak ridership did visibly grow, but not to levels that congest the system.</p>
<p>But then the hourly regional trains got completely overcrowded. If you wanted to ride the free trains from Berlin to Leipzig, you’d be standing for the last third of the trip. This is because the regional rail system (<a href="https://pedestrianobservations.com/2019/08/20/s-bahn-and-regionalbahn/">as opposed to S-Bahn</a>) is designed as a low-capacity coverage-type system for connecting to small towns like Cottbus or Dessau.</p>
<p>The broader issue is that there is always a sharp ridership gradient between large cities and everywhere else, even per capita. In some places the gradient is sharper than elsewhere; the difference between New York and the rest of the United States is massive. But even in Germany, with a smaller gradient than one might be used to from France or the UK or Japan, public transport ridership is disproportionately dense urban or perhaps suburban, on trams and U- and S-Bahns.</p>
<p>The regional trains are another world. Really, European and Japanese trains can be thought of as three worlds: very high-use urban and suburban rail networks, high-use intercity rail connecting the main cities usually at high speed, and low-usage, highly-subsidized regional trains outside the major metropolitan regions. Germany has relatively good trains in the last category, if worse than in Switzerland, Austria, or the Netherlands: they run hourly with timed connections, so that people can connect between them to many destinations, they just usually don’t because cities the size of Dessau don’t generate a lot of ridership. The 9€ ticket gave people a free intercity trip if they chained trips on these regional trains, at the cost of getting to Leipzig in a little less than three hours rather than 1:15 on the ICE; the regional trains were not expanded to meet this surge traffic, which is usually handled on longer intercity trainsets, creating standing-room only conditions on trains where this should not happen off-peak or perhaps ever.</p>
<p><strong>The issue of fare integration</strong></p>
<p>The overcrowding seen on the regional trains last summer is really an issue of fare integration, which I hope is resolved as the 49€ gives people free trips on such trains permanently. A cornerstone of good public transport planning is that the fare between two points should be the same no matter what vehicle one uses, with exceptions only for first-class cars if available. <em>Ein Ticket für alles</em>, exclaims the system in Zurich, to great success. Anything else slices the market into lower-frequency segments, providing worse service than under total fare integration. Germany understands this – the Verkehrsverbund was invented in Hamburg in 1965, and subsequently this idea was adopted elsewhere until <a href="https://commons.wikimedia.org/wiki/File:Karte_der_Verkehrsverb%C3%BCnde_und_Tarifverb%C3%BCnde_in_Deutschland.png">the country has been divided into metropolitan zones with internal fare integration</a>.</p>
<p>The regional trains that cross Verkehrsverbund zones have their own fares, and normally that’s okay. Intercity trains were never part of this system, and that’s okay too – they’re not about one’s usual trip, and so an intercity ticket doesn’t include free transfers to local public transport unless one pays extra for that amenity. The fares between intercity trains and chains of regional trains were not supposed to be integrated, and normally that’s fine too, because any fare savings from chaining trips on slower trains are swamped both by the headache of buying so many tickets and by the difference in trip time and reliability.</p>
<p>The 9€ ticket broke that system, and the 49€ ticket will have the same effect: for three months, trips on slower trains were free, leading to overcrowding on a low-capacity network that normally isn’t that important to the country’s overall public transport system.</p>
<p>Worse, the operating costs of slow trains are higher than those of fast trains: they are smaller and so have a higher ratio of crew to passengers than ICEs, and their slowness means that crew and maintenance costs per kilometer are higher than those of fast trains. <a href="https://journals.sagepub.com/doi/abs/10.3141/2159-04">Even energy costs are higher on slow trains</a>, because high-speed lines run at 300 km/h over long stretches, whereas regional lines make many stops (which had very little usage compared with the train’s volume of passengers last summer) and have slow zones rather than cruising at 130 or 160 km/h over long stretches. So the system gave people a price incentive to use the higher-cost trains and not the lower-cost ones.</p>
<p>This is the most important thing to resolve about any future fare reductions. Some mechanism is needed to ensure that the most advantageous way to travel between two cities is the one that DB can provide the most efficiently, which is IC/ICE and not RegionalBahn.</p>
	</div>
	</article>
	<article>
	<header>
	<time>2023-01-25</time>
	<h2>
	<a href="https://pedestrianobservations.com/2023/01/25/our-construction-costs-report/">
	Our Construction Costs Report	</a>
	</h2>
	</header>
	<div>
	
<p>This is awkward.</p>
<p>I was hoping to already release the Transit Costs Project report, but we ran into difficulties after <a href="https://pedestrianobservations.com/2022/10/24/the-transit-costs-project-conclusion-is-out/">we released</a> and then immediately unreleased; we sent the report to the MTA for feedback, we got feedback, and then we went on a binge of edits. This led to a long cycle of almosts; I thought it would be done at the end of December, then 10 days ago, then tomorrow. The revisions were not huge – there are people who read the original version and the gap between what they’ve seen and what we’re about to release is real but is also far from the difference between complete knowledge (which we don’t have and most likely never will) and complete ignorance.</p>
<p>We still have minor edits to do, but they’re small, on the order of less than a full day’s work for all three of us, and we can promise that the full report – including the New York report and the synthesis of all reports combined – will be out on <strong>February 6th</strong>.</p>
<p>Separately, I’m going to be in New York for about a month after, mostly for non-work reasons. The timing is that <a href="https://en.wikipedia.org/wiki/2023_Berlin_repeat_state_election">there’s an election in Berlin on the 12th</a>, a redo of the city election from 2021 (oddly, not electing for a full term but for the remainder of the term that began in 2021), in which I hope the Greens do better if only so that they’re slightly ahead of SPD and not slightly behind and can replace the scandalized Franziska Giffey as mayor. If I can vote by mail, I’ll be in New York starting a little before the election; if I can’t, I’ll be starting a little later. This will include at least one more in-person event to present our findings and our recommendations for the region and the MTA, at a time to be determined but likely in the second half of February.</p>
<p>There’s a lot of stuff we didn’t get into the report, because of lack of space; we could have info-dumped and made the synthesis novel-length, but some things had to go (and I will defend every decision to exclude things, and if they turn out to have been necessary, then blame me for being wrong about it). So even if you read the reports you should definitely come to events and ask me specific questions about things that are related to construction costs, such as other projects in the same cities we studied, or issues of the interplay between costs and benefits, or how this applies to urban rail construction going forward.</p>
	</div>
	</article>
	<article>
	<header>
	<time>2023-01-24</time>
	<h2>
	<a href="https://pedestrianobservations.com/2023/01/24/push-and-pull-factors-and-measuring-modal-shift/">
	Push and Pull Factors and Measuring Modal Shift	</a>
	</h2>
	</header>
	<div>
	
<p>There’s a longstanding debate among activists and academics about what the best way of effecting modal shift from cars to public transport is. Pull factors concern making public transport better through building more rail lines, running them more frequently, improving service convenience, or reducing fares. Push factors concern making driving harder through speed limits, fuel taxes, congestion pricing, and reallocation of street space from cars to public and non-motorized transport. There’s a tendency on the New Left to favor push factors (but the East Asian developmental states are best characterized as push-before-pull and not pure pull).</p>
<p>This has been refined by researchers at the climate research institute, the <a href="https://ariadneprojekt.de/">Ariadne Project</a>, who published a paper in late 2021 rating various push and pull policies on effectiveness for reducing transport emissions. <a href="https://ariadneprojekt.de/media/2021/10/Ariadne-Analyse_KlimaschutzundVerkehr_Oktober21.pdf">They conclude that push factors dominate, and pull factors are small</a>, with construction of new public transit almost insignificant, only worth a reduction of around 300,000 tons of CO2 a year Germany-wide, 0.039% of national emissions as of 2021; instituting a 120 km/h speed limit on the Autobahn is said to have about 10 times that effect, while the biggest effects yet would come from carbon taxes. The study laments that pull factors are so much more popular than push factors, which they admit suppress society-wide consumption.</p>
<p>The research suffers from the same problem as other work in this direction, in that it is bad at estimating the impact of public transport on mode shift. It briefly argues that construction of public transport increases overall consumption and therefore doesn’t do much to reduce emissions. This way, it’s like <a href="https://www.tagesspiegel.de/berlin/gutachter-stellen-katastrophale-co2-bilanz-fur-neue-tunnel-auf-8108782.html">2020’s carbon critique of U-Bahn expansion</a>, <a href="https://pedestrianobservations.com/2022/11/21/berlin-greens-know-the-price-of-everything-and-value-of-nothing/">which I criticized two months ago</a>; the carbon critique argues that each kilometer of U-Bahn built only reduces CO2 emissions by 714 tons a year through mode shift, under the assumption that only 20% of public transport riders are diverted from cars.</p>
<p>This doesn’t pass a sanity check. 300,000 divided by 714 is 420 km, which is about comparable to the total route length of the four grade-separated U-Bahn systems in Germany plus the Wuppertal Schwebebahn; I think the two figures, 300,000 and 714/km, come from different sources, and judging by the other elements in the study, I suspect 300,000 assumes less construction than a full doubling of Germany’s rapid transit network length. Nonetheless, even under a more generous assumption, this is far too low compared with macro trends in public transport usage.</p>
<p>The best way to use macro trends as a sanity check is to look at some cases with much more and much less public transport than the present. Do they look like it’s a total difference of 0.039%? No, and that’s even taking into account that transit cities tend to be wealthier, stimulating more consumption and more production. As I pointed out in my post two months ago, while Germany averages 9.15 t-CO2/capita, Berlin only does 5.38, and while Germany averages 580 cars per 1,000 people, Berlin only does 327. The difference is largely about Berlin’s pull factors. Push factors in the city are not extensive, and what exists is implemented only in areas that already have very low car use.</p>
<p>Even lower household emissions in Berlin must be viewed as downstream of the density that is enabled by the presence of a large urban rail network. Cars are a low-capacity mode of transport, so an auto-oriented region, like American metro regions, has to spread out its homes and destinations to limit congestion, and this increases household emissions (single-family houses emit more than apartment buildings) and also encourages people to travel longer distances for their commute and routine non-commute trips.</p>
<p>This is not easy to measure. Public transport projects have gotten fairly good in the last generation at estimating ridership, but estimating the responsibility of one particular project to modal shift is hard. It interacts with the entire city region. For example, building one rail line can be measured to shift modes in the neighborhoods it serves, but it also encourages destinations to locate in city center since people from the neighborhoods the line serves can now access it, and the increase in office, retail, and community development then leads to a small modal shift citywide. Worse, trying to tease out the effect of the rail line on modal shift sufficiently carefully may lead researchers to count this citywide effect negatively, since one econometric technique is to compare the neighborhoods near the line with neighborhoods in the same city not on the line.</p>
<p>In practice, the construction of rail lines tends to co-occur with other policies that improve public transport, which may be pull or push factors. This means that it’s very easy to misattribute the effect of urban rail expansion to those other factors. I am convinced that this is what is happening here; the proper comparison must be at the level of an entire region, looking at the emissions of different regions with different levels of public transport usage.</p>
<p>The upshot is that if it is hard to measure the effect of public transport construction on modal shift and emissions, then the uncertain factors should not be set to zero. Rather, they should be set to sanity-check levels. For example, one can compare New York with the rest of the United States, since it’s a starker difference between a transit city and an auto-oriented country than anywhere in Europe, and correct for non-transport effects like climate and electricity mix, both of which are easy to measure.</p>
<p>Within Germany, Berlin has 42% lower emissions than the rest of the country per capita. Berlin achieves this with an urban rail network that, in 2019, got 1,289 million rail trips, nearly all within the city of 3.7 million, a minority in the suburban region of perhaps 1.3 million. This is around 250 trips/person regionwide, and 320/person citywide assigning around 20% of S-Bahn ridership to suburbs like Potsdam and Oranienburg. What’s more, Germany doesn’t start from zero; this is not the United States, with multiple large cities with around 10 annual rail trips per capita. Netting out buses from <a href="https://www.vdv.de/vdv-statistik-2020.pdfx">VDV’s data</a> (p. 25) gets around 6.3 billion rail trips in Germany in 2019 including trams, or 75 per capita.</p>
<p>The difference between 320 and 75 is around 250 – I know it’s actually 245 but at this point I’m deliberately reducing precision because those are sanity-check estimates and I don’t want people thinking they’re correct to three significant figures (try 1.5). If we attribute the entire Berlin-Germany difference of about 3.8 t-CO2/capita to public transport and downstream changes to the urban layout, then we get 0.015 t saved per annual trip generated. To get from there to 300,000 tons saved, we just need 20 million annual rail riders, or around 65,000 daily ones, which is easy to generate on a single line; <a href="https://www.tagesspiegel.de/berlin/das-nein-zur-verlangerung-der-u8-ist-falsch-5097508.html">the approximately 2 km extension of U8 to Märkisches Viertel that Berlin keeps postponing is estimated to generate 25,000-30,000</a>.</p>
<p>Now, to sanity-check the sanity check, the estimate here is that every trip on urban rail saves 15 kg-CO2. This is an aggressive figure; new cars nowadays average 100 g/km and averaged 180 g/km in 2001 (<a href="https://eupocketbook.org/wp-content/uploads/2023/01/Pocketbook_2022_23_Web.pdf">source</a>, PDF-p. 15), and the average displaced car trip is not 150 km or even 80 km – Americans average around 45 km/day, or somewhat more when only adults are considered. Rather, the issue is a combination of factors:</p>
<ul>
<li>Because the limiting factor to car transport is capacity, in practice what happens in an auto-oriented region is that it fills from the inside outward, and any modal shift ends up displacing the outermost and longest car trips. <a href="https://pedestrianobservations.com/2019/01/15/public-transit-is-greener-than-it-seems/">I proposed a model for that in a blog post from four years ago</a>.</li>
<li>Public transport displaces car trips on a more than one-to-one basis (and certainly more than 20% as in the carbon critique of the U-Bahn). This is because public transport users also walk and bike, and transit cities have high modal splits for active transport by the standards of auto-oriented cities, if not by the standards of Dutch cities. <a href="https://www.berlin.de/sen/uvk/presse/pressemitteilungen/2020/pressemitteilung.906382.php">Berlin’s all-trip modal split in 2018</a> was 26% car, 27% public transport, 18% bike, 30% walking – and the high active transport modal split exists not because of road diets, which are few and far between, but because of the presence of a large core fed by the U- and S-Bahn.</li>
<li>Public transport reduces household energy usage by encouraging people to live in apartment buildings with shared walls rather than in single-family houses, which have much greater heating requirements; this is also the mechanism through which transit cities have relatively high usage of active transport even without trying very hard.</li>
</ul>
<p>I don’t think these factors fully explain away the gap between 45 km/day and 150 km per trip (so around 300/day), but they explain a large enough fraction of it that the installation of a system like what Berlin has – or, better, what Tokyo has – should be a climate priority. If your model says it doesn’t, it needs a lot more work than to just talk about the consumption effects of more public transport (if you’re bothered by how Berlin is poor for its size, compare New York with the rest of the United States).</p>
<p>In fact, if estimating modal shift is hard, then it’s best to approximate it by ridership. It’s imperfect because there is the effect of walking and biking; some lines really do just compete with walking, like city-center streetcars, but usually, to first order, it’s a good enough estimate. If it’s hard to estimate the benefits then they should not be set to zero, but rather set proportionally to something easier to measure, in this case ridership. Investment should follow ridership-maximizing strategies, and only deviate from them in corner cases.</p>
	</div>
	</article>
	<article>
	<header>
	<time>2023-01-22</time>
	<h2>
	<a href="https://pedestrianobservations.com/2023/01/22/paris-berlin-trains-but-no-infrastructure/">
	Paris-Berlin Trains, But no Infrastructure	</a>
	</h2>
	</header>
	<div>
	
<p>Yesterday, <a href="https://www.bloomberg.com/news/articles/2023-01-22/france-germany-plan-high-speed-train-between-paris-and-berlin">Bloomberg reported that Macron and Scholz announced new train service between Paris and Berlin</a> to debut next year, as intercity rail demand in Europe is steadily rising and people want to travel not just within countries but also between them. Currently, there is no direct rail service, and passengers who wish to travel on this city pair have to change trains in Frankfurt or Cologne. There’s just one problem: the train will not have any supportive infrastructure and therefore take the same eight hours that trains take today with a transfer.</p>
<p>This is especially frustrating, since Germany is already investing in improving its intercity rail. Unfortunately, the investments are halting and partial – right now the longest city pair connected entirely by high-speed rail is Cologne-Frankfurt, a distance of 180 km, and ongoing plans are going to close some low-speed gaps elsewhere in the system but still not create any long-range continuous high-speed rail corridor connecting major cities. With ongoing plans, Cologne-Stuttgart is going to be entirely fast, but not that fast – <a href="https://www.frankfurt-mannheim.de/home.html">Frankfurt-Mannheim</a> is supposed to be sped up to 29 minutes over about 75 km.</p>
<p>Berlin-Paris is a good axis for such investment. This includes the following sections:</p>
<ul>
<li>Berlin-Halle is currently medium-speed, trains taking 1:08-1:16 to do 162 km, but the flat, low-density terrain is easy for high-speed rail, which could speed this up to 40-45 minutes at fairly low cost since no tunnels and little bridging would be required.</li>
<li>Halle-Erfurt is already fast, thanks to investments in the Berlin-Munich axis.</li>
<li>Erfurt-Frankfurt is currently slow, but there are <a href="https://de.wikipedia.org/wiki/Aus-_und_Neubaustrecke_Hanau%E2%80%93W%C3%BCrzburg/Fulda%E2%80%93Erfurt#2010er_Jahre">plans to build high-speed rail from Erfurt to Fulda and thence Hanau</a>. The trip times leave a lot to be desired, but newer 300 km/h trains like the Velaro Novo, and perhaps a commitment to push the line not just to Hanau but closer to Frankfurt itself, could do this section in an hour.</li>
<li>Frankfurt-Saarbrücken is very slow. Saarbrücken is at the western margin of Germany and is not significant enough by itself to merit any high-speed rail investment. Between it and Frankfurt, the terrain is rolling and some tunneling is needed, and the only significant intermediate stops are Mainz (close enough to Frankfurt it’s a mere stop of opportunity) and Kaiserslautern. Nonetheless, fast trains could get from Frankfurt to the border in 45 minutes, whereas today they take two hours.</li>
</ul>
<p>Unfortunately, they’re not talking about any pan-European infrastructure here. Building things is too difficult, so instead the plan is to run night trains – this despite the fact that Frankfurt-Saarbrücken with a connection to the LGV Est would make a great joint project.</p>
	</div>
	</article>
	<article>
	<header>
	<time>2023-01-19</time>
	<h2>
	<a href="https://pedestrianobservations.com/2023/01/19/bad-public-transit-in-the-third-world/">
	Bad Public Transit in the Third World	</a>
	</h2>
	</header>
	<div>
	
<p>There’s sometimes a stereotype that in poor countries with low car ownership, alternatives to the car are flourishing. I saw <a href="https://mastodon.social/@Pkbwood@mastodon.online/109702911922586851">a post on Mastodon making this premise</a>, and pointed out already in comments that this is not really true. This is a more detailed version of what I said in 500 characters. In short, in most of the third world, non-car transportation is bad, and nearly all ridership (on jitneys and buses) is out of poverty, as is most walking. While car ownership is low, the elites who do own cars dominate local affairs, and therefore cities are car-dominated and not at all walkable, even as 90%+ of the population does not own a car.</p>
<p>What’s more, the developing countries that do manage to build good public transportation don’t stay developing for long. The same development model of Japan, the East Asian Tigers, and now China has built both rail-oriented cities and high economic growth, to the point that Japan and the Tigers are fully developed, and China is a solidly middle-income economy. The sort of places that stay poor, or get stuck in a middle-income trap, also tend to have stagnant urban rail networks, and so grow more auto-oriented over time. </p>
<p><strong>The situation in Southeast Asia</strong></p>
<p>With the exception of Singapore, nowhere in Southeast Asia is public transit good. What’s more, construction costs have been high for elevated lines and very high for underground ones, slowing down the construction of metro systems.</p>
<p>In Kuala Lumpur and Bangkok, motorization is high and public transit usage is weak. <a href="https://www.reinventingtransport.org/2010/11/download-my-thesis-on-urban-transport.html">Paul Barter’s thesis</a> details how both cities got this way, in comparison with the more transit-oriented model used in Tokyo, Seoul, Hong Kong, and Singapore. The thesis also predicts that the poorer megacities of Southeast Asia – Jakarta and Manila – will follow the auto-oriented path as they develop, which has indeed happened in the 13 years since it was written.</p>
<p>The situation in those cities is, to be fair, murky. Manila has a large urban rail under construction right now, with average to above average costs for elevated lines and high ones for subways. But the system it has today consists of four lines, two branded light rail, one branded MRT, and one commuter line. <a href="https://www.bworldonline.com/economy/2021/11/15/410827/rail-ridership-seen-taking-at-least-3-years-to-return-to-normal-levels/">In 2019, the six-month ridership on the system was 162 million</a>. A total of 324 million in a metro area the size of Manila is extraordinarily low: the administrative Metro Manila region has 13.5 million people, and the urban or metropolitan area according to both Citypopulation.de and Demographia is 24-26 million. On the strictest definition of Metro Manila, this is 24 trips per person per year; on the wider ones, it is about 13, similar to San Diego or Portland and only somewhat better than Atlanta.</p>
<p>Jakarta is in the same situation of flux. It recently opened a half-underground MRT line at fairly high cost, and is modernizing its commuter rail network along Japanese lines, using second-hand Japanese equipment. <a href="https://megapolitan.kompas.com/read/2022/06/19/18455611/pengguna-krl-jabodetabek-disebut-capai-12-juta-orang-per-hari?page=all#:~:text=JAKARTA%2C%20KOMPAS.com%20%2D%20Menteri,19%2F6%2F2022)">Commuter rail ridership was 1.2 million a day last year</a>, or around 360 million a year, already higher than before corona; <a href="https://en.antaranews.com/news/269043/jakarta-mrt-ridership-reaches-197-million-in-2022">the MRT had 20 million riders last year</a>, and <a href="https://economy.okezone.com/read/2019/01/10/320/2002648/ka-bandara-soetta-sepi-penumpang-ini-alasan-kai">an airport link had 1.5 million in 2018</a>. This isn’t everything – there’s also a short light metro called LRT for which I can’t find numbers – but it wouldn’t be more than second-order. This is 400 million annual rail trips, in a region of 32 million people.</p>
<p>The future of these cities is larger versions of Bangkok. Thailand is sufficiently middle-income that we can see directly how its transport system evolves as it leaves poverty, and the results are not good. Bus ridership is high, but it’s rapidly falling as anyone who can afford a car gets one; a <a href="https://economy.okezone.com/read/2019/01/10/320/2002648/ka-bandara-soetta-sepi-penumpang-ini-alasan-kai">JICA report about MRT development</a> puts the region’s modal split at 5% MRT, 36% bus, and the rest private (PDF-p. 69) – and the income of bus riders is significantly lower than that of drivers (PDF-p. 229), whereas MRT riders are closer to drivers.</p>
<p>Even wealthier than Bangkok, with the same auto-oriented system, is Kuala Lumpur. <a href="https://www.researchgate.net/figure/Current-modal-split-versus-targeted-modal-split-for-2020-adapted-by-Chiu-Chuen-et-al_fig9_320000866">There, the modal split is about 8% bus, 7% train, and the rest private</a>. This is worse than San Francisco and the major cities of Canada and Australia, let alone New York or any large European city. The national modal split in England, France, Germany, and Spain is about 16% – the first three countries’ figures predate corona, but <a href="https://www.ine.es/jaxi/Tabla.htm?tpx=55359&amp;L=0">in Spain they’re from 2021</a>, with suppressed public transport ridership. Note that rail ridership per capita is healthier in Kuala Lumpur than in Jakarta or Manila – <a href="https://www.unescap.org/sites/default/files/6.%20SUTI_Malaysia.pdf">all rail lines combined are 760,000 riders per day</a>, say 228 million per year, in a region of maybe 7 million people. This is better than a no-transit American city like San Diego, but worse than a bad-transit one like Chicago or Washington, where the modal split is about the same but there is no longer the kind of poverty that is common in Malaysia, let alone in Indonesia, and therefore if people ride the trains it’s because they get them to their city center jobs and <a href="https://www.ine.es/jaxi/Tabla.htm?tpx=55359&amp;L=0">not because they’re poor</a>.</p>
<p>Even in Singapore, the best example out there of a transit-oriented rich city, it took until very recently for MRT coverage to be good enough that people willingly depend on it; it only reached NUS after I graduated. In the 1990s, the epitome of middle-class Singaporean materialism was described as owning the <a href="https://en.wikipedia.org/wiki/Five_Cs_of_Singapore">Five Cs</a>, of which one was a car; traffic suppression, a Paul Barter describes, has centered fees on cars, much more car purchase than car use (despite the world-famous congestion pricing system), and thus to those wealthy enough to afford cars, they’re convenient in ways they are not in Paris, Berlin, or Stockholm.</p>
<p><strong>The situation in Africa</strong></p>
<p>African countries between the Sahara and the Kalahari are all very poor, with low car ownership. However, they are thoroughly car-dominated.</p>
<p>From the outside, it’s fascinating to see how the better-off countries in that region, like Nigeria, are already imitating Southeast Asia. Malaysia overregulated its jitneys out of existence because they were messy and this bothered elites, and because it wanted to create an internal market for its state-owned automakers. Nigeria is doing the same, on the former grounds; to the extent <a href="https://guardian.ng/news/agbero-lagos-drivers-still-pay-multiple-charges-despite-harmonisation/">it hasn’t happened</a> despite <a href="https://guardian.ng/news/nigeria/can-chairman-wants-danfo-buses-banned-from-lagos-roads/">years of trying</a>, it’s because the state is too weak to do more than harass the drivers and users of the system.</p>
<p>It’s notable that the Lagos discourse about the evils of the danfo – they are noisy, they are polluting, they drive like maniacs – there is little attention to how cars create all the same problems, except at larger scale per passenger served. The local notables drive (or are driven); the people who they scorn as unwashed, overly fecund, criminal masses ride the danfo. Thanks to aggressive domination by cars and inattention to the needs of the non-driving majority, Lagos’s car ownership is high for how poor it is – <a href="https://lagosstate.gov.ng/blog/2017/07/05/lagos-and-motor-vehicle-administration/">one source from 2017</a> says 5 million cars in the state, <a href="https://www.carmart.ng/public/blog/50-of-13-million-vehicles-on-nigerian-roads-are-in-lagos-and-kano/">another from 2021</a> says 6.5 million vehicles between the state and Kano State. The denominator population in the latter source is 27 million officially, but unofficially likely more; 200 vehicles per 1,000 people is plausible for Lagos, which to be clear is not much less than New York or Paris, on an order of magnitude lower GDP per capita. Tokyo took until about 1970 to reach 100 vehicles per 1,000 people, at which point Japan had almost fully converged with American GDP per capita.</p>
<p>This is not specific to Lagos. A cousin who spent some time in Kampala told me of the hierarchy on the roads: pedestrians fear motorcycles, motorcycles fear cars, cars fear trucks. There is no pedestrian infrastructure to speak of; a rapid transit system is still a dream, to the point that <a href="https://techjaja.com/dreamland-what-if-kampala-had-an-underground-metro-system/">a crayon proposal that spread on Twitter made local media</a>. That the vast majority of Ugandans don’t own cars doesn’t matter; Kampala remains dominated by the few who do.</p>
<p><strong>Transit and development</strong></p>
<p>I don’t think it’s a coincidence that the sort of developing countries that build successful urban rail systems don’t stay poor for long. Part of it is that public transportation is good for economic development, but that’s not most of it – the United States manages to be rich without it except in a handful of cities. Rather, I suspect the reason has to do with state capacity.</p>
<p>More specifically, the reason cities with 100-200 cars per 1,000 people are thoroughly dominated by cars is that those 10-20% drivers (or people who are driven) are the elites. Their elite status can come from any source – passive business income, landlordism, active business income, skilled professional work – but usually it tilts toward the traditional, i.e. passive. These groups tend to be incredibly anti-developmental: they own small businesses, sometimes actively and sometimes passively, and resent being made redundant through economies of scale. India has problems with <a href="https://www.dhyeyaias.com/current-affairs/daily-current-affairs/indias-economy-and-the-challenge-of-informality">economic dwarfism and informality</a>, and this is typical of poor countries; if anything, India is better than most at developing a handful of big businesses in high-value added industries.</p>
<p>The upshot is that the sort of people who drive, and especially the sort of drivers who are powerful enough to effect local changes to get incremental upgrades to roads at the expense of non-drivers, are usually anti-developmental classes. The East Asian developmental states (and Singapore and Hong Kong, which share many characteristics with them) clamped down on such classes hard, on either nationalist or socialist grounds; Japan, both Koreas, and both Chinas engaged in land reform, with characteristic violence in the two socialist states and without it but still with forcible purchase in the three capitalist states. The same sort of state that can eliminate landlordism can also, as a matter of capital formation, clamp down on consumption and encourage personal savings, producing atypically low levels of motorization well into middle-income status. Singapore, whose elite consumption centers vacations out of the country, has managed to do so even as a high-income country – and even more normal Tokyo and Seoul have much higher rail usage and lower car usage than their closest Western analog, New York.</p>
<p>India is in many ways anti-developmental, but it does manage to grow. <a href="https://pedestrianobservations.com/2022/08/09/growth-without-urbanization/">Its anti-developmentalism is anti-urban and NIMBY</a>, but it is capable of building infrastructure. Its metro program has problems with high construction costs (but Southeast Asia’s are generally worse) and lack of integration with other modes such as commuter rail, which the middle class denigrates as only befitting poor people; but <a href="https://timesofindia.indiatimes.com/city/delhi/metro-ridership-hits-87-of-pre-pandemic-levels/articleshow/94951303.cms">the Delhi Metro had 5.5 million daily riders just before corona</a>, slightly behind New York in a slightly larger metro area, perhaps a better comparison than Jakarta and Manila’s San Diego.</p>
<p>It’s the slower-growing developing countries that are not managing to even build the systems India has, let alone East Asia. They don’t have high car use, but only because they are poor, and in practice, they are thoroughly car-dominated, and everyone who doesn’t have a car wants one. A rich country really is not one where even the poor have cars but where even the rich use public transportation – and those countries aren’t rich and don’t grow at rates that will make them rich.</p>
	</div>
	</article>
	<article>
	<header>
	<time>2023-01-12</time>
	<h2>
	<a href="https://pedestrianobservations.com/2023/01/12/high-costs-are-not-about-precarity/">
	High Costs are not About Precarity	</a>
	</h2>
	</header>
	<div>
	
<p>I’ve seen people who I think highly of argue that high construction costs in the United States are an artifact of precarity. The argument goes that the political support for public transportation there is so flimsy that agencies are forced to buy political support by spending more money than they need. This may include giving in to NIMBY pressure to use costlier but less impactful (or apparently less impactful) techniques, to spread money around with other government agencies and avoid fighting back, to build extravagant and fancier-looking but less standardized stations, and so on. The solution, per this theory, is to politically support public transportation construction more so that transit agencies will have more backing.</p>
<p>This argument also happens to be completely false, and the solution suggested is counterproductive. In fact, the worst cost blowouts are for the politically most certain projects; Second Avenue Subway enjoyed unanimous support in New York politics.</p>
<p><strong>Cost-effectiveness under precarity</strong></p>
<p>Three projects relevant to our work at the Transit Costs Project have been done exceptionally cost-effectively in an environment of political uncertainty: the T-bana, the LGV Sud-Est, and Bahn 2000.</p>
<p><em>T-bana</em></p>
<p>The original construction of the T-bana was done at exceptionally low cost. We go over this in the <a href="https://transitcosts.com/city/sweden-case/">Sweden report</a> to some extent, but, in short, between the 1950s and 70s, the total cost of the system’s construction was 5 billion kronor in 1975 prices, which built around 100 km, of which 57% are underground. In PPP 2022 dollars, this is $3.6 billion, or $35 million/km, not entirely but mostly underground. This was low for the time: for example, in London, the Victoria line was $122 million/km and the Jubilee line was $172 million/km (<a href="http://www.omegacentre.bartlett.ucl.ac.uk/wp-content/uploads/2014/12/UK_JLE_PROFILE.pdf">source</a>, p. 78), and <a href="https://transitcosts.com/city/italy/">Italian costs in the 1960s and 70s were similar</a>, averaging $129 million/km before 1970.</p>
<p>The era of Social Democrat dominance in Swedish politics on hindsight looks like one of consensus in favor of big public projects. But the T-bana itself was controversial. When the decision was made to build it in the 1940s, Stockholm County had about 1 million people; at the time, metros were present in much larger cities, like New York, London, Paris, Berlin, and Tokyo, and it was uncertain that a city the size of Stockholm would need such a system. Its closest analog, Copenhagen, did not build such a system until the 1990s, when it was a metro region of 2 million. It was uncertain that Stockholm should need rapid transit, and there were arguments for and against it in the city. Nor was there any transit-first policy in postwar Sweden: urban planning was the same modernist combination of urban renewal, automobile scale, and tower-in-a-park housing, and outside Stockholm County, the Million Program projects were thoroughly car-oriented.</p>
<p>Construction costs in Sweden are a lot higher now than they were in the 1950s, 60s, and 70s. Nya Tunnelbanan is $230 million/km, compared with a post-1990s Italian average of $220 million; British costs have exploded in tandem, so that now the Underground extensions clock at $600 million/km. Our best explanation is that the UK adopted what we call the <em>globalized system</em> of procurement, privatizing planning functions to consultants and privatizing risk to contractors, which creates more conflict; the UK also has an unusually high soft cost factor. From American data (and not just New York) and some British data, I believe that the roughly 2.5 cost premium of the UK over Italy is entirely reducible to such soft costs, procurement conflict, risk compensation, and excessive contingency. And yet, Sweden itself, with some elements of the same globalized system, maintains a roughly Italian cost level, albeit trending the wrong way.</p>
<p>And today, too, the politics of rail expansion in Sweden are uncertain. There was controversy over both Citybanan and Nya Tunnelbanan, neither of which passed a cost-benefit analysis (for reasons that I believe impugn the cost-benefit analysis more than those projects); it was uncertain that either would be funded. Controversy remained over plans to build high-speed rail connecting Stockholm with Gothenburg and Malmö, and the newly-elected right-wing government just canceled them in order to prioritize investment in roads. Swedish rail projects today remain precarious, and have to justify themselves on cost and efficiency grounds.</p>
<p><em>LGV Sud-Est</em></p>
<p>Like nearly all other rich countries, France was hit hard by the 1973 oil crisis; economic growth there and in the US, Japan, and most of the rest of Western Europe would never be as high as it was between the end of WW2 and the 1970s (“<a href="https://en.wikipedia.org/wiki/Trente_Glorieuses">Trente Glorieuses</a>“). On hindsight, France’s response to the crisis models can-go governance, with an <a href="https://www.youtube.com/watch?v=aHoJ130hQ3E">energy saving ad</a> declaring “in France we don’t have oil, but we have ideas.” The French state built nuclear power plants with gusto, peaking around 90% of national electricity use – and even today’s reduced share, around 70%, is by a large margin the highest in the world. At the same time, it built a high-speed rail network, connecting Paris with most other provincial cities at some of the highest average speeds outside China between major cities, reaching about 230 km/h between Paris and Marseille and 245 km/h between Paris and Bordeaux; usage per capita is one of the highest in Europe and, measured in passenger-km, not too bad by East Asian standards.</p>
<p>But in fact, the first LGV, the LGV Sud-Est, was deeply controversial. At the time, the only high-speed rail network in operation was the Shinkansen, and while France learned more from Japan than any other European country (for example, <a href="https://www.ejrcf.or.jp/jrtr/jrtr23/pdf/F36_Sato.pdf">the RER was influenced by Tokyo rail operations</a>), the circumstances for intercity were completely different. SNCF had benefited from having done many of its own experiments with high-speed technology, but the business case was murky. SNCF had to innovate in running an open system, with extensive through-running to cities off the line, which Japan would only introduce in the 1990s with the Mini-Shinkansen.</p>
<p>Within the French state, the project was controversial. Anthony Perl’s New Departures details how there were people within the government who wanted to cancel it entirely as it was unaffordable. At the end, the French state didn’t finance the line, and required SNCF to find private loans on the international market, though it did guarantee those loans. It also delayed the line’s opening: instead of opening the entire line from Paris to Lyon in one go, it opened two-thirds of it on the Lyon side in 1981 and the last third into Paris in 1983, requiring trains to run on the classical line at low speed between Paris and Saint-Florentin for two years; in that era, phased opening was uncommon, and lines generally opened to the end at once, such as between Tokyo and Shin-Osaka.</p>
<p>Construction was extraordinarily inexpensive. In PPP 2022 dollars, it cost $8.4 million/km. This is, by a margin, the lowest-cost high-speed rail line ever built that I know about. The <a href="half a century of speed - International Railway Journal">Tokaido Shinkansen cost </a><a href="https://www.railjournal.com/in_depth/shinkansen-half-a-century-of-speed/">380</a><a href="half a century of speed - International Railway Journal"> billion yen</a>, or in PPP 2022 dollars $40 million/km, representing a factor of two cost overrun that forced JNR’s head to resign. Spain has unusually low construction costs, and even there, Madrid-Seville was $15.7 million/km. SNCF innovated in every way possible to save money. Realizing that high-speed trains could climb steeper grades, it built the LGV Sud-Est with a ruling grade of 3.5%, which has since become a norm in and around Europe, compared with the Shinkansen’s 1.5-2%; the line has no tunnels, unlike the classical Paris-Lyon line. It built the line on the ground rather than on viaducts, and balanced cut and fill locally so that material cut to grade the line could be used for nearby fill. Thanks to the line’s low costs and high ridership, the financial return on investment for SNCF has been 15%, and social return on investment has been 30% (<a href="https://www.cade-environnement.org/files/2014/03/bilanrentabilitegpso.pdf">source</a>, pp. 11-12).</p>
<p>This cost-effectiveness would never recur. The line’s success ensured that LGV construction would enjoy total political backing. The core features of LGV construction are still there – earthworks rather than viaducts, 3.5% grades, limited tunneling, overcompensation of landowners by about 30% with land swap deals to defuse the possibility of farmer riots. But the next few lines cost about $20 million/km or slightly less, and this cost has since crept up to about $30 million/km or even more. This remains low by international standards (but not by Spanish ones), but the trend is negative.</p>
<p>SNCF is coasting on its success from a generation ago, secure that funding for LGVs and state support in political contention is forthcoming, and the routing decisions have grown worse. In response to NIMBYism in Provence, the French state assented to a tunnel-heavy route, including a conversion of Marseille from an at-grade terminal to an underground through-station, akin to <a href="https://en.wikipedia.org/wiki/Stuttgart_21">Stuttgart 21</a>, which has not been done before in France, and the resulting high costs have led to delays on the project. Operations have grown ever more airline-style, experimenting with low-cost airline imitation to the point of reducing fare receipts without any increase in ridership. One of the French consultants we’ve spoken with said that their company’s third-party design costs are 7-8% of the hard costs, which figure is similar to what we’ve seen in Italy and to the in-house rate in Spain – but the same consultant told us that there is so much bloat at SNCF that when it designs its own projects, the costs are not 7% but 25%, a figure in line with American rates.</p>
<p><em>Bahn 2000</em></p>
<p>Switzerland has Europe’s strongest passenger rail network by all measures: highest traffic measured by passenger-km per capita, highest modal split for passenger-km, highest traffic density. Its success is well-known in surrounding countries, which are gradually either imitate its methods or, in the case of Germany, pretending to do so. It has achieved its success through continuous improvement over the generations, but the most notable element of this system was implemented in the 1990s as part of the Bahn 2000 project.</p>
<p>The current system is based on a national-scale clockface system (“Takt”) with trains repeating hourly, with the strongest links, like the Zurich-Bern-Basel triangle, running every half hour. Connections are timed in those three cities and several others, called knots, so that trains enter each station a few minutes before the connection time (usually the hour) and depart a few minutes after, permitting passengers to get between most pairs of Swiss cities with short transfers. Reliability is high, thanks to targeted investments designed to ensure that trains could make those connections in practice and not just in theory. Further planning centers adding more knots and expanding this system to the periphery of Switzerland.</p>
<p>Switzerland is famous for its consensus governance system – its plural executive is drawn from the four largest parties in proportion to their votes, with no coalition vs. opposition politics. But the process that led to the decision to adopt Bahn 2000 was not at all one of unanimity. There had been plans to build high-speed rail, as there were nearly everywhere else in Western Europe. But they were criticized for their high costs, and there was extensive center-right pressure to cut the budget. Bahn 2000 was thus conceived in an environment of austerity. Many of its features were explicitly about saving money:</p>
<ul>
<li>The knot system is connected with running trains as fast as necessary, not as fast as possible. Investments in speed are pursued only insofar as they permit trains to make their connections; higher speeds are considered gratuitous.</li>
<li>Bilevel trains are an alternative to lengthening the platforms.</li>
<li>Timed overtakes and meets are an alternative to more extensive multi-tracking of lines.</li>
<li>Investment in better timetabling and systems (the electronics side of the electronics-before-concrete slogan) is cheaper than adding tunnels and viaducts.</li>
</ul>
<p>Swiss megaprojects have to go to referendum, and sometimes the referendums return a no; this happened with the Zurich U-Bahn twice, leading to the construction of the S-Bahn instead. All Swiss planners know in a country this small and this fiscally conservative, any extravagance will lead to rejection. The result is that they’ve instead optimized construction at all levels, and even their unit costs of tunneling are low; thanks to such optimization, Switzerland has been able to build a fairly extensive medium-speed rail system, with more tunneling per capita than Germany (let alone France), and with two S-Bahn trunk tunnels in Zurich, where no German city today has more than one.</p>
<p><strong>The American situation</strong></p>
<p>The worst offenders in the United States are not at all politically precarious. There is practically unanimous consensus in New York about the necessity of Second Avenue Subway. At no point was the project under any threat. There is an ideological right in the city, rooted less in party politics and more in the New York Post and the Manhattan Institute, with a law-and-order agenda and hostility to unions and to large government programs, but at no point did they call for cancellation; the Manhattan Institute’s Nicole Gelinas has proposed pension cuts for workers and rule changes reducing certain benefits, but not canceling Second Avenue Subway.</p>
<p>At intercity scale, the same is true of Northeast Corridor investment. The libertarian and conservative pundits who say passenger rail is a waste of money tend to except the Northeast Corridor, or at least its southern half. When the Republicans won the 2010 midterm election, the new chair of the House of Representatives Transportation Committee, John Mica (R-FL), proposed a bill to seek private concessionaires to run intercity rail on the corridor. He did not propose canceling train service, even though in the wake of the same election, multiple conservative governors canceled intercity rail investments in their state, both high-speed (Florida) and low-speed (Wisconsin, Ohio).</p>
<p>In fact, both programs – New York subway expansion and the Northeast Corridor – are characterized by continuity across partisan shifts, as in more established consensus governance systems. The Northeast Corridor is especially notable for how little role ideological or partisan politics has played so far. New York has micromanagement by politicians – Andrew Cuomo had his pet projects in Penn Station Access and the backward-facing LaGuardia air train, and now Kathy Hochul has hers in the Interborough Express – but Second Avenue Subway was internal, and besides, political micromanagement is a different problem from political precarity.</p>
<p>And neither of these programs has engaged in any cost control. To the contrary, both are run as if money is infinite. <a href="https://pedestrianobservations.com/2022/11/26/its-easy-to-waste-money/">The MTA would surrender to NIMBYs (“good neighbor policy”) and to city agencies looking to extract money from it</a>. It built oversize stations. It spent money protecting buildings from excessive settlement that have been subsequently demolished for redevelopment at higher density.</p>
<p>The various agencies involved in the Northeast Corridor, likewise, are profligate, and not for lack of political support. Connecticut is full of NIMBYs; one of the consultants working on the plan a few years ago told me there was informal pressure not to ruffle feathers and not to touch anything in the wealthiest suburbs in the state, those of Fairfield County. In fact, <a href="https://pedestrianobservations.com/2018/01/12/high-speed-rail-from-new-rochelle-to-greens-farms-impacts-opportunities-and-analysis/">high-speed rail construction would require significant house demolitions in the state’s second wealthiest town, Darien</a> – but Darien is so infamously exclusive (“Darien rhymes with Aryan,” say other suburbanites in the area) that the rest of the region feels little solidarity with it.</p>
<p>NIMBYs aside, there has not been any effort at coordinating the different agencies in the Northeast along anything resembling the Swiss Takt system. This is not about precarity, because this is not a precarious project; this is about total ignorance and incuriosity about best practices, which emanate from a place that doesn’t natively speak English and doesn’t trade in American political references.</p>
<p><em>The Green Line Extension</em></p>
<p>Boston’s GLX is a fascinating example of cost blowouts without precarity. The history of the project is that its first iteration was pushed by Governor Deval Patrick (D), with the support of groups that sued the state for its delays in planning the project in the 1990s, as a court-mandated mitigation for the extra car traffic induced by the construction of the Big Dig. Patrick instituted a good neighbor policy, in which everything a community group wanted, it would get. Thus, stations were to become neighborhood signatures, and the project was laden with unrelated investments, called betterments, like a $100 million 3 km bike lane called the Somerville Community Path.</p>
<p>At no point in the eight years Patrick was in power was there a political threat to the project. It was court-mandated, and the extractive local groups that live off of suing the government favored it. The Obama administration was generous with federal stimulus funding, and the designs were rushed in order to use stimulus funding to pay for the project’s design, which would be done by consultants rather than with an expansion of in-house hiring. It’s in this atmosphere of profligacy that the project’s cost exploded to, in today’s money, around $3.5 billion, for 7 km of light rail in existing rights-of-way (albeit ones requiring overpass reconstruction).</p>
<p>The project did fall under political threat, after Charlie Baker (R) won the 2014 election. Baker’s impact on Massachusetts governance is fascinating, in that he unambiguously cut its budget significantly in the short run, but also had both before (as budget director in the 1990s) and during (through his actions as governor) wrecked the state’s long-term ability to execute infrastructure, setting up a machine intended to privatize the state and avoiding any in-house hiring. Nonetheless, the direct impact of precarity on GLX was to reduce scope: the betterments were removed and the stations were changed from too big to too small. The final cost was $2.3 billion, or around $2.8 billion in 2022 dollars, and half of that was sunk cot from the previous iteration.</p>
<p>There was no expectation that the project would be canceled – indeed, it was not. A Republican victory was unexpected in a state this left-wing. Then, as Baker was taking office, past governors from both parties expressed optimism that he would get not just GLX done but also the much more complex North-South Rail Link tunnel. Nor did contractors have their contracts yanked unexpectedly, which would get them to bid higher for risk compensation. Baker cold-shouldered not jut NSRL but also much smaller investments in commuter rail, but at no point was anyone stiffed in the process. No: the Patrick-era project was just poorly managed.</p>
<p><em>Project selection</em></p>
<p>As one caveat, I want to point out a place where precarity does lead to poor project cost-effectiveness: the project selection stage, in the context of tax referendums, especially in Southern California. None of this leads to high per-kilometer costs – Los Angeles’s are exactly as bad as in the rest of the United States – but it does lead to poor project selection, as an unintended consequence of anti-tax New Right politics.</p>
<p>The California-specific issue is that raising taxes requires a referendum, with a two-thirds vote. Swiss referendums are by simple majority, or at most double majority – but in practice most referendums that have a popular vote majority, even a small one, also have a double majority when needed, and in the last 15 years I believe the only two times they didn’t had the double majority veto overturning a <a href="https://en.wikipedia.org/wiki/2020_Swiss_referendums#November_referendums">1.5% margin</a> and a <a href="https://en.wikipedia.org/wiki/2013_Swiss_referendums#March_referendums">9% margin</a>.</p>
<p>California’s requirement of a two-thirds majority was intended to stop wasteful spending and taxation, but has had the opposite effect. In and around San Francisco, the voters are sufficiently left-wing that two-thirds majorities for social policy are not hard to obtain. But in Southern California, they are not; to build public transportation infrastructure, Los Angeles and San Diego County governments cannot rely on ideology, but instead cut deals with local, non-ideological actors through promising them a piece of the package. Los Angeles measures for transit expansion are, by share of money committed, largely not about transit expansion but operations, new buses, or leakage to roads and bridges; then, what does go to transit expansion is divvied by region, with each region getting something, no matter how cost-ineffective, while core improvements are neglected and so are cross-regional connections (since the local extractive actors aren’t going to ride the trains and can’t tell a circumferential project is useful for them).</p>
<p>This is not a US-wide phenomenon. It’s not even California-wide: this problem is absent from the Bay Area, where BART decided against an expansion to Livermore that was unpopular among technically-oriented advocates, and would like to build more core capacity if it could do so for much less than a billion dollars per km. New York does not have it at all (for one, it doesn’t require two-thirds majorities for budgeting).</p>
<p>At intercity scale, this precarity does cause Amtrak to maximize how many states and congressional districts it runs money-losing daily trains in. But wasting money on night trains and on peripheral regions is hardly a US-only problem – Japan National Railways did that until well past privatization, when its successors spun off money-losing branch lines to prefecture-subsidized third sector railways. This is not at all why there is no plan for Northeastern intercity rail that is worth its weight in dirt: Northeastern rail improvements have been amply funded relative to objective need (if not relative to American costs), and solid investments in the core coexist with wastes of money on the periphery of the network in many countries.</p>
<p><strong>The issue of politicization</strong></p>
<p>The precarious, low-cost examples all had to cut costs because of fiscal pressure. However, in all three, the pressure did not include any politicization of engineering questions. Sweden was setting up a civil service modeled on the American Bureau of Public Roads, currently the Federal Highway Administration, which in the middle of the 20th century was a model of depoliticized governance. France and Switzerland have strong civil service bureaucracies – if anything SNCF is too self-contained and needs reorganization, just not if it’s led by the usual French elites or by people from the airline industry.</p>
<p>Importantly, low-cost countries with more clientelism and politicization of the state tend to be more deferential to the expertise of engineers. Greece has a far worse problem of overreliance on political appointees than the United States, let alone the other European democracies; but engineering is somewhat of an exception. Hispanic and Portuguese-speaking cultures put great prestige on engineering, reducing the extent of political micromanagement, even in countries without strong apolitical civil service bureaucracies. Even in Turkey, the politicization of public transportation is entirely at macro level: AKP promises to prioritize investment in areas that vote for it and has denied financing to the Istanbul Metro since the city flipped to the opposition (the city instead borrows money from the European Investment Bank), but below that level there is no micromanagement.</p>
<p>The American examples, in contrast, show much more political micromanagement. This is part of the same package as the privatization of state planning in the globalized system; in the United States often there was never the depoliticization that most of the rest of the developed and middle-income world had, but on top of that, the tendency has been to shut down in-house planning departments or radically shrink them and replace them with consultants. The consultants are then supervised by political appointees with no real qualifications to head capital programs, and the remaining civil servants are browbeaten not to disagree with the political appointees’ proclamations.</p>
<p>Those political appointees rarely measure themselves by any criteria of infrastructure utility. Even in New York they and the managers don’t consistently use the system; in Los Angeles, they use it about as often as the executive director of a well-endowed charity eats at a soup kitchen. To them, the cost is itself a measure of success – and this is true of other agencies, which treat obtaining other people’s money as a mark of achievement and as testament to their power. This behavior then cascades to local advocacy groups, which try to push solutions that maximize outside funding and are at bet indifferent and at worst actively hostile to any attempt at efficiency.</p>
<p>Just giving more support to agencies in their current configuration is not going to help. To the contrary, it only confirms that profligacy gets rewarded. A program of depoliticization of the state is required in tandem with expanding in-house hiring and reversing the globalized system, and the political appointees and the managers and political advocates who are used to dealing with them don’t belong in this program.</p>
	</div>
	</article>
	<article>
	<header>
	<time>2023-01-07</time>
	<h2>
	<a href="https://pedestrianobservations.com/2023/01/07/edge-cities-with-and-without-historic-cores/">
	Edge Cities With and Without Historic Cores	</a>
	</h2>
	</header>
	<div>
	
<p>An edge city is a dense, auto-oriented job center arising from nearby suburban areas, usually without top-down planning. The office parks of Silicon Valley are one such example: the area had a surplus of land and gradually became the core of the American tech industry. In American urbanism, Tysons in Virginia is a common archetype: the area was a minor crossroads until the Capital Beltway made it unusually accessible by car, providing extensive auto-oriented density with little historic core.</p>
<p>But there’s a peculiarity, I think mainly in the suburbs of New York. Unlike archetypal edge cities like Silicon Valley, Tysons, Century City in Los Angeles, or Route 128 north of Boston, some of the edge cities of New York are based on historic cores. Those include White Plains and Stamford, which have had booms in high-end jobs in the last 50 years due to job sprawl, but also Mineola, Tarrytown, and even New Brunswick and Morristown.</p>
<p>The upshot is that it’s much easier to connect these edge cities to public transportation than is typical. In Boston, I’ve spent a lot of time trying to figure out good last mile connections from commuter rail stations. Getting buses to connect outlying residential areas and shopping centers to town center stations is not too hard, but then Route 128 is completely unviable without some major redesign of its road network: the office parks front the freeway in a way that makes it impossible to run buses except dedicated shuttles from one office park to the station, which could never be frequent enough for all-day service. Tysons is investing enormous effort in sprawl repair, which only works because the Washington Metro could be extended there with multiple stations. Far and away, these edge cities are the most difficult case for transit revival for major employment centers.</p>
<p>And in New York, because so much edge city activity is close to historic cores, this is far easier. Stamford and White Plains already have nontrivial if very small transit usage among their workers, usually reverse-commuters who live in New York and take Metro-North. Mineola could too if the LIRR ran reverse-peak service, but it’s about to start doing so. Tarrytown and Sleepy Hollow could be transit-accessible. The New Jersey edge cities are harder – Edison and Woodbridge have lower job density than Downtown Stamford and Downtown White Plains – but there are some office parks that could be made walkable from the train stations.</p>
<p>I don’t know what the history of this peculiar feature is. White Plains and Mineola are both county seats and accreted jobs based on their status as early urban centers in regions that boomed with suburban sprawl in the middle of the 20th century. Tarrytown happened to be the landfall of the Tappan Zee Bridge. Perhaps this is what let them develop into edge cities even while having a much older urban history than Tysons (a decidedly non-urban crossroads until the Beltway was built), Route 128, or Silicon Valley (where San Jose was a latecomer to the tech industry).</p>
<p>What’s true is that all of these edge cities, while fairly close to train stations, are auto-oriented. They’re transit-<em>adjacent</em> but not transit-<em>oriented</em>, in the following ways:</p>
<ul>
<li>The high-rise office buildings are within walking distance to the train station, but not with a neat density gradient in which the highest development intensity is nearest the station.</li>
<li>The land use at the stations is parking garages for the use of commuters who drive to the station and use the train as a shuttle from a parking lot to Manhattan, rather than as public transportation the way subway riders do.</li>
<li>The streets are fairly hostile to pedestrians, featuring fast car traffic and difficult crossing, without any of the walkability features that city centers have developed in the last 50 years.</li>
</ul>
<p>The street changes required are fairly subtle. Let us compare White Plains with <a href="https://en.wikipedia.org/wiki/Metrotown,_Burnaby">Metrotown</a>, both image grabs taken from the same altitude:</p>
<figure><a href="https://pedestrianobservations.files.wordpress.com/2023/01/whiteplains.png"></a></figure>
<figure><a href="https://pedestrianobservations.files.wordpress.com/2023/01/metrotown.png"></a></figure>
<p>These are both edge cities featuring a train station, big buildings, and wide roads. But in Metrotown, the big buildings are next to the train station, and the flat-looking building to its north is the <a href="https://en.wikipedia.org/wiki/Metropolis_at_Metrotown">third-largest shopping mall in Canada</a>. The parking goes behind the buildings, with some lots adjoining Kingsway, which has a frequent trolleybus (line 19) but is secondary as a transportation artery to SkyTrain. Farther away, the residential density remains high, with many high-rises in the typical thin-and-tall style of Vancouver. In contrast, in White Plains, one side of the station is a freeway with low-density residential development behind it, and the other is parking garages with office buildings behind them instead of the reverse.</p>
<p>The work required to fix this situation is not extensive. Parking must be removed and replaced with tall buildings, which can be commercial or residential depending on demand. This can be done as part of a transit-first strategy at the municipal level, but can also be compelled top-down if the city objects, since the MTA (and other Northeastern state agencies) has preemption power over local zoning on land it owns, including parking lots and garages.</p>
<p>On the transit side, the usual reforms for improvements in suburban trains and buses would automatically make this viable: high local frequency, integrated bus-rail timetables (to replace the lost parking), integrated fares, etc. The primary target for such reforms is completely different – it’s urban and inner-suburban rail riders – but the beauty of the S-Bahn or RER concept is that it scales well for extending the same high quality of service to the suburbs.</p>
	</div>
	</article>
	<article>
	<header>
	<time>2022-12-30</time>
	<h2>
	<a href="https://pedestrianobservations.com/2022/12/30/the-four-quadrants-of-cities-for-transit-revival/">
	The Four Quadrants of Cities for Transit Revival	</a>
	</h2>
	</header>
	<div>
	
<p>Cities that wish to improve their public transportation access and usage are in a bind. Unless they’re already very transit-oriented, they have not only an entrenched economic elite that drives (for example, small business owners almost universally drive), but also have a physical layout that isn’t easy to retrofit even if there is political consensus for modal shift. Thus, to shift travel away from cars, new interventions are needed. Here, there is a distinction between old and new cities. Old cities usually have cores that can be made transit-oriented relatively easily; new cities have demand for new growth, which can be channeled into transit-oriented development. Thus, usually, in both kinds of cities, a considerably degree of modal shift is in fact possible.</p>
<p>However, it’s perhaps best to treat the features of old and new cities separately. The features of old cities that make transit revival possible, that is the presence of a historic core, and those of new cities, that is demand for future growth, are not in perfect negative correlation. In fact, I’m not sure they consistently have negative correlation at all. So this is really a two-by-two diagram, producing four quadrants of potential transit cities.</p>
<p><strong>Old cities</strong></p>
<p>The history of public transportation is one of decline in the second half of the 20th century in places that were already rich then; newly-industrialized countries often have different histories. The upshot is that an old auto-oriented place must have been a sizable city before the decline of mass transit, giving it a large core to work from. This core is typically fairly walkable and dense, so transit revival would start from there.</p>
<p>The most successful examples I know of involve the restoration of historic railroads as modern regional lines. Germany is full of small towns that have done so; <a href="https://zierke.com/">Hans-Joachim Zierke</a> has some examples of <a href="https://zierke.com/shasta_route/pages/44reactivate.html">low-cost restoration of regional lines</a>. Overall, Germany writ large must be viewed as such an example: while German economic growth is healthy, population growth is anemic, and the <a href="https://www.mobilitaet-in-deutschland.de/pdf/infas_Mobilitaet_in_Deutschland_2017_Kurzreport_DS.pdf">gradual increase in the modal split for public transportation here</a> must be viewed as more intensive reuse of a historic national rail network, anchored by tens of small city cores.</p>
<p>At the level of a metropolitan area, the best candidates for such a revival are similarly old places; in North America, the best I can think of for this are Philadelphia, Boston, and Chicago. Americans don’t perceive any of the three as especially auto-oriented, but their modal splits are comparable to those of small French cities. But in a way, they show one way forward. If there’s a walkable, transit-oriented core, then it may be attractive for people to live near city center; in those three cities it’s also possible to live farther away and commute by subway, but in smaller ones (say, smaller New England cities), the subway is not available but conversely it’s usually affordable to live within walking distance of the historic city center. This creates a New Left-flavored transit revival in that it begins with the dense city center as a locus of consumption, and only then, as a critical mass of people lives there, as a place that it’s worth building new urban rail to.</p>
<p><strong>New cities</strong></p>
<p>Usually, if a city has a lot of recent growth from the era in which it has become taken for granted that mobility is by car, then it should have demand for further growth in the future. This demand can be planned around growth zones with a combination of higher residential density and higher job density near rail corridors. The best time to do transit-oriented development is before auto-oriented development patterns even set in.</p>
<p>There are multiple North American examples of how this works. The best is Vancouver, a metropolitan area that has gone from 560,000 people in the 1951 census to 2.6 million in the 2021 census. Ordinarily, one should expect such a region to be entirely auto-oriented, as most American cities with almost entirely postwar growth are; but in 2016, the last census before corona, it had a 20% work trip modal split, and that was before the Evergreen extension opened.</p>
<p>Vancouver has achieved this by using its strong demand for growth to build a high-rise city center, with office towers in the very center and residential ones ringing it, as well as high-density residential neighborhoods next to the Expo Line stations. The biggest suburbs of Vancouver have followed the same plan: Burnaby built an entirely new city center at Metrotown in conjunction with the Expo Line, and even more auto-oriented Surrey has built up Whalley, at the current outer terminal of the line, as one of its main city centers. Housing growth in the region is rapid; YIMBY advocacy calls for more, but the main focus isn’t on broad development (since this already happens) but on permitting more housing in recalcitrant rich areas, led by the West Side, which will soon have its Broadway extension of the Millennium Line.</p>
<p>Less certain but still interesting examples of the same principle are Calgary, Seattle, and Washington. Calgary, a low-density city, planned its growth around the C-Train, and built a high-rise city center, limiting job sprawl even as residential sprawl is extensive; Seattle and the Virginia-side suburbs of Washington have permitted extensive infill housing and this has helped their urban rail systems achieve high ridership by American standards, Seattle even overtaking Philadelphia’s modal split.</p>
<p><strong>The four quadrants</strong></p>
<p>The above contrast of old and new cities misses cities that have positive features of both – or neither. The cities with both positive features have the easiest time improving their public transportation systems, and many have never been truly auto-oriented, such as New York or Berlin, to the point that they’re not the best examples to use for how a more auto-oriented city can redevelop as a transit city.</p>
<p>In North America, the best example of both is San Francisco, which simultaneously is an old city with a high-density core and a place with immense demand for growth fueled by the tech industry. The third-generation tech firms – those founded from the mid-2000s onward (Facebook is in a way the last second-generation firm, which generation began with Apple and Microsoft) – have generally headquartered in the city and not in Silicon Valley. Twitter, Uber, Lyft, Airbnb, Dropbox, and Slack are all in the city, and the traditional central business district has expanded to South of Market to accommodate. This is really a combination of the consumption-oriented old-city model, as growing numbers of employees of older second-generation firms chose to live in the city and reverse-commute to Silicon Valley, and the growth-oriented new-city model. Not for nothing, the narrower metropolitan statistical area of San Francisco (without Silicon Valley) reached a modal split of 17% just before corona, the second highest in the United States, with healthy projections for growth.</p>
<p>But then there is the other quadrant, comprising cities that have neither the positive features of old cities nor those of new cities. To be in this quadrant, a city must not be so old as to have a large historic core or an extensive legacy rail network that can be revived, but also be too poor and stagnant to generate new growth demand. Such a city therefore must have grown in a fairly narrow period of time in the early- to mid-20th century. The best example I can think of is Detroit. The consumption-centric model of old city growth can work even there, but it can’t scale well, since there’s not enough of a core compared with the current extent of the population to build out of.</p>
	</div>
	</article>
	</section>||||I|||| Pedestrian Observations

 For Walkability and Good Transit, and Against Boondoggles and Pollution
 * About
 * Comment Policy
 * Construction Costs
 * Post Queue
 * Support Me!
 2023-01-30

 FDP and Vice Signaling

 Finance Minister Christian Lindner (FDP) just tweeted that more investment in roads is good – because if traffic flows more smoothly then there will be less greenhouse gas emissions. Reaction was not positive, and as of when I’m writing, 16 hours later, it is mildly ratioed. People understand that this is wrong. Lindner himself probably gets this too. Understanding what’s going on here requires talking about bullshit in the philosophical sense of Harry Frankfurt, and about something that I don’t have a better name for than vice signaling.

 Is it true?

 Absolutely not. It’s standard in transport studies that the construction of more highways in high-demand areas induces more traffic, as people take advantage of the greater convenience of driving. Drivers drive to new destinations that they forwent or chose to take public transport to, and new developments are built in areas opened by new highway development.

 There may be exceptions to this in declining areas. The United States loves building new grade-separated interchanges in declining regions. This doesn’t generate new demand, because traffic is already uncongested, and the purpose of roadbuilding there is a political statement more than transport policy. But that’s not Germany. The roads under discussion here are in growth regions: there’s a plan to widen the beltway around Munich, A99, to 10 lanes, and the federal and Berlin FDP have both badgered Berlin to build a further stage of A100 parallel to the Ringbahn, which the city wants not to under the influence of the Green Party. Both motorway projects are likely to lead to adverse mode shift if built, and Lindner knows this.

 There’s a developmental argument that induced demand is actually good. Matt Yglesias has made it before, saying that if road building induces more traffic then it means people get to take more trips and are better off. Many roadbuilders have made that very argument, and others were aware of it; Robert Moses, for example, was perfectly aware that his parkways and bridges were inducing more car traffic, and was fine with it, because he thought more driving was good. But that’s not what Lindner is saying: Lindner is saying that building new motorways and keeping them without a speed limit reduces greenhouse gas emissions, which is just bullshit.

 Bullshit

 The term “bullshit” has a precise meaning in analytic philosophy, due to Harry Frankfurt. It comprises a type of deception about the speaker’s mindset, rather than about the facts, unlike an ordinary lie. A politician who denies a scandal they are involved with is lying: their goal is to get you to believe that they are innocent of this scandal. A politician who, having been caught in said scandal, launches a series of schlock patriotic speeches is bullshitting: their goal is to get you to think they are fundamentally aligned with your values. From Frankfurt’s original essay, we have,

 Telling a lie is an act with a sharp focus. It is designed to insert a particular falsehood at a specific point in a set or system of beliefs, in order to avoid the consequences of having that point occupied by the truth. This requires a degree of craftsmanship, in which the teller of the lie submits to objective constraints imposed by what he takes to be the truth. The liar is inescapably concerned with truth-values. In order to invent a lie at all, he must think he knows what is true. And in order to invent an effective lie, he must design his falsehood under the guidance of that truth. On the other hand, a person who undertakes to bullshit his way through has much more freedom. His focus is panoramic rather than particular. He does not limit himself to inserting a certain falsehood at a specific point, and thus he is not constrained by the truths surrounding that point or intersecting it. He is prepared to fake the context as well, so far as need requires. This freedom from the constraints to which the liar must submit does not necessarily mean, of course, that his task is easier than the task of the liar. But the mode of creativity upon which it relies is less analytical and less deliberative than that which is mobilized in lying. It is more expansive and independent, with mare spacious opportunities for improvisation, color, and imaginative play. This is less a matter of craft than of art. Hence the familiar notion of the “bullshit artist.”

 The statement “widening roads reduces CO2 emissions” is this kind of bullshit. It is not quite a lie: it is false, but Lindner is not especially concerned with whether it is true or false. His goal is not to persuade people that building another section of A100 and widening A99 is good for climate; nobody who cares about climate change thinks that. Rather, his goal is to position himself as the sort of person who doesn’t listen to climate advocates and will just push for road widenings. The deception is part of the positioning: if he’d said that he understands the Greens’ argument against road investment but roads are important for economic development, he’d come off as too reasonable, which is not his intention.

 Sounding deliberately unreasonable is the domain of populist politicians, and Frankfurt himself and many of his followers have noticed how political bullshit is on the rise as populism grows more normalized. Nigel Farage, for example, bullshitted that smoking isn’t bad for your health. And FDP is a populist party, despite its liberal origins and relatively moderate political positioning; it swung from deficit scold at the start of the current government to tax scold precisely as inflation rose last year, the opposite of what one should expect of a Washington Consensus-following economically orthodox party.

 Vice signaling

 There’s a pseudo-academic term going around the web, virtue signaling. The idea is that individuals and organizations engage in actions to signal that they’re better people than they really are; companies hire consultants on diversity, equity, and inclusion (DEI) without ever doing anything about their glass ceiling and harassment problems.

 But it may be more fruitful to discuss its opposite – that is, vice signaling. This is when people take actions to portray themselves as terrible people, for any number of reasons:

 * Loyalty: criminal gangs are deliberately threatening and often require that prospective members commit murder (this is a requirement to become a made man in the Italian-American mafia), because this forces new members to have crossed both a moral and a legal event horizon from which they can’t come back; populist political movements don’t require crimes, but do require ridiculous beliefs
 * Novelty: this is what in the online language of the early 2010s was called the Slate Pitch – a take that aims to be novel by saying something really out there, often by writers who can’t separate themselves from the rest of the pack by any more productive means
 * Love of power: some people lie to you, with your full knowledge that they’re lying, just to flex that they can get away with it

 Lindner loves this kind of vice signaling, I think out of novelty more than anything. FDP could be a party of YIMBYism, fiscal conservatism, and digital governance; younger members of the party who identify with neoliberalism wish that it were that party. The problem is that the difference between such a party and SPD is not large; Scholz ran on building more housing Germany-wide, and there’s a fair amount of consensus in favor of this in the party’s wings. SPD’s worst attributes so far are its officious leadership anchored in the Lower Saxony clique and consequently its sluggish governance and refusal to do more to support Ukraine – but FDP has the exact same problems, Lindner having told Ukraine when it asked for aid as the war started that there was no point since they’d fall in hours either way.

 So to distinguish themselves from everyone else, FDP engages in vice signaling about climate and transport. They’re not trying to convince anyone that their policies are good for climate change. Rather, they’re doing the exact opposite: they’re trying to convince center-right voters that they’re an internal opposition within a coalition that is engaging in modal shift in federal funding priorities, and that they are explicitly against any climate action, because cars are good and only annoying hippies prefer trains.

 Written by Alon Levy 56 Comments Posted in Cars , Environmental Issues , Germany , Politics and Society , Transportation
 2023-01-28

 Quick Note: High Speed 2 and Euston

 There was reporting in the Sun, since officially denied, that Britain is planning to cut Euston from its high-speed rail project and run trains only as far a Old Oak Common, a future development site west of Central London. I assume given the source and lack of any other confirmation that the plans are to run to Euston as planned. But what if the story is not completely fake news, and there are plans to cut on construction at Euston? I can see a cut being positive value engineering, using space at the station more efficiently.

 What’s the issue with High Speed 2?

 High Speed 2 is an extremely expensive line. Among proper intercity high-speed rail lines (as opposed to suburban lines running at medium speed), it is the most expensive in our database per kilometer. The projected cost as of 2019-20 is about the same as that of all lines built to date in Germany and France combined; Germany has about 1,000 km of newly-built high-speed lines and France 2,500, whereas HS2 is planned to be 530 km.

 The high costs are related to some massive scope creep, including tunnels in relatively flat terrain through the Chilterns, dug essentially because the area has rich NIMBYs and the British state decided not to fight them. Those are already in advanced enough construction that I don’t think descoping them and building the line at-grade with compulsory purchase of land is viable. However, some of the scope is new stations, which British defenders of the system insist are necessary. Birmingham is to get an entirely new station at Curzon Street, and London Euston is to get a substantial increase in size, with additional tracks and approaches. This is said to be necessary for capacity reasons.

 Are new stations necessary for capacity?

 No. Euston today has 16 platform tracks; it had 18 before HS2 construction started. The S-Bahn-quality Watford DC line can use two; the remaining slow services at the station amount to around 10-12 trains per hour, which S-Bahn-quality terminals like Saint-Lazare on the RER E and Catalunya on the Barcelona Rodalies network can comfortably turn on four tracks; those two comparisons turn 16 and 24 trains per hour on four tracks, respectively. The services out of Euston branch more than the RER and Rodalies, but this is mostly a mix of stopping patterns, largely on the same legacy line.

 Then there’s HS2 itself. The line is expected to get very high ridership, justifiably: all cities along the line are larger than their comparison cases on the LGV Sud-Est, often substantially, and the projection is that very high capacity is required, on the order of 16 trains per hour. This stretches high-speed rail to its limit: the Shinkansen, which mixes local and express trains on double track, peaks around 14-15 trains per hour, and the complexly branched TGV around 12. HS2 expects to do better perhaps through better signals but also through having a simpler stopping pattern on its most congested section, between London and the bifurcation at Birmingham Interchange, on which trains are to run nonstop.

 However, 16 trains per hour can still turn on about six platform tracks. This is not easy: the Tohoku Shinkansen turns 14 on four tracks, but this is a limit case, famous not just in rail media but also in business media as successful optimization of infrastructure. Nonetheless, given how constrained the site is, it’s useful to learn from the best and not the average. If it’s possible to descope the plans to add new tracks to Euston, this should be done; present plans for Euston cost billions.

 Is this happening?

 Maybe. Britain is aware of the situation at Tokyo Station, although it seems more interested in looking for reasons not to learn than in learning. Perhaps very high costs are leading to a reevaluation, in which Euston can be made smaller than in current plans and trains can turn more efficiently.

 But again, the ultimate source on this said nothing of this sort, and is unreliable. So who knows?

 Written by Alon Levy 46 Comments Posted in High-Speed Rail , Transportation
 2023-01-26

 Free Public Transport, Fare Integration, and Capacity

 There’s an ongoing debate about free public transport that I’m going to get into later, but, for now, I want to zoom in on one aspect of the 9€ ticket, and how it impacted public transport capacity in Germany. A commenter on the Neoliberal Reddit group claimed that during the three months of nearly free public transport fares, there was a capacity crunch due to overuse. But in fact, the impact was not actually significant on urban rail, only on regional trains, in a way that underscores the importance of fare integration more than anything.

 What was the 9€ ticket?

 Last year, in the wake of the Russian invasion of Ukraine, fuel prices shot up everywhere. This created populist pressure to alleviate the price of fuel through temporary tax cuts, which further exacerbated last year’s high inflation. The center-right element within the German coalition, FDP, moved away from its traditional position as deficit scold and demanded a cut in the fuel tax; as a compromise, the Greens agreed to it on condition that during the three months of reduced fuel tax, June through August, public transport fares would be cut as well. Thus the 9€ monthly was born.

 The 9€ ticket applied throughout Germany. The key feature wasn’t just the deep discount but also the fact that on one ticket, people could travel all over Germany; normally, my Berlin monthly doesn’t let me ride the local trains in Leipzig or Munich. This stimulated massive domestic tourism, since people could travel between cities on slow regional trains for free and then also travel around their destination city for free as well.

 What now?

 The 9€ ticket clearly raised public transport ridership in the three months it was in effect. This led to demands to make it permanent, running up against the problem that money is scarce and in Germany ticket fares generate a significant proportion of public transport revenue, 7.363 billion € out of 14.248 billion € in expenses (source, p. 36).

 One partial move in that direction is a 29€ monthly valid only within Berlin, not in the suburbs (zone C of the S-Bahn) or outside the system; unlike the 9€ ticket, which was well-advertised all over national and local media and was available at every ticketing machine, the 29€ monthly is only available via annual subscription, which requires a permanent address in the city, and the regular machines only sell the usual 86€ monthly and don’t even let you know that a cheaper option exists. The subscription is also not available on a rolling basis – one must do it before the start of the month, which is not advertised, and Ant6n‘s family was caught unaware one month.

 Negotiations for a nationwide 49€ ticket are underway, proceeding at the pace of a German train, or perhaps that of German arms deliveries to Ukraine. This was supposed to start at the beginning of 2023, then in April, and now it’s expected to debut in May. I’m assuming it will eventually happen – German trains get you there eventually, if hours late occasionally.

 What’s the impact on capacity?

 The U- and S-Bahn systems didn’t at all get overcrowded. They got a bit more crowded than usual, but nothing especially bad, since the sort of trips induced by zero marginal cost are off-peak. Rush hour commuters are not usually price-sensitive: whenever one’s alternative to the train is a car, the difference between a 9€ monthly and an 86€ one is a fraction of the difference between either ticket and the cost of owning and using a car, and at rush hour, cars are limited by congestion as well. Off-peak ridership did visibly grow, but not to levels that congest the system.

 But then the hourly regional trains got completely overcrowded. If you wanted to ride the free trains from Berlin to Leipzig, you’d be standing for the last third of the trip. This is because the regional rail system (as opposed to S-Bahn) is designed as a low-capacity coverage-type system for connecting to small towns like Cottbus or Dessau.

 The broader issue is that there is always a sharp ridership gradient between large cities and everywhere else, even per capita. In some places the gradient is sharper than elsewhere; the difference between New York and the rest of the United States is massive. But even in Germany, with a smaller gradient than one might be used to from France or the UK or Japan, public transport ridership is disproportionately dense urban or perhaps suburban, on trams and U- and S-Bahns.

 The regional trains are another world. Really, European and Japanese trains can be thought of as three worlds: very high-use urban and suburban rail networks, high-use intercity rail connecting the main cities usually at high speed, and low-usage, highly-subsidized regional trains outside the major metropolitan regions. Germany has relatively good trains in the last category, if worse than in Switzerland, Austria, or the Netherlands: they run hourly with timed connections, so that people can connect between them to many destinations, they just usually don’t because cities the size of Dessau don’t generate a lot of ridership. The 9€ ticket gave people a free intercity trip if they chained trips on these regional trains, at the cost of getting to Leipzig in a little less than three hours rather than 1:15 on the ICE; the regional trains were not expanded to meet this surge traffic, which is usually handled on longer intercity trainsets, creating standing-room only conditions on trains where this should not happen off-peak or perhaps ever.

 The issue of fare integration

 The overcrowding seen on the regional trains last summer is really an issue of fare integration, which I hope is resolved as the 49€ gives people free trips on such trains permanently. A cornerstone of good public transport planning is that the fare between two points should be the same no matter what vehicle one uses, with exceptions only for first-class cars if available. Ein Ticket für alles, exclaims the system in Zurich, to great success. Anything else slices the market into lower-frequency segments, providing worse service than under total fare integration. Germany understands this – the Verkehrsverbund was invented in Hamburg in 1965, and subsequently this idea was adopted elsewhere until the country has been divided into metropolitan zones with internal fare integration.

 The regional trains that cross Verkehrsverbund zones have their own fares, and normally that’s okay. Intercity trains were never part of this system, and that’s okay too – they’re not about one’s usual trip, and so an intercity ticket doesn’t include free transfers to local public transport unless one pays extra for that amenity. The fares between intercity trains and chains of regional trains were not supposed to be integrated, and normally that’s fine too, because any fare savings from chaining trips on slower trains are swamped both by the headache of buying so many tickets and by the difference in trip time and reliability.

 The 9€ ticket broke that system, and the 49€ ticket will have the same effect: for three months, trips on slower trains were free, leading to overcrowding on a low-capacity network that normally isn’t that important to the country’s overall public transport system.

 Worse, the operating costs of slow trains are higher than those of fast trains: they are smaller and so have a higher ratio of crew to passengers than ICEs, and their slowness means that crew and maintenance costs per kilometer are higher than those of fast trains. Even energy costs are higher on slow trains, because high-speed lines run at 300 km/h over long stretches, whereas regional lines make many stops (which had very little usage compared with the train’s volume of passengers last summer) and have slow zones rather than cruising at 130 or 160 km/h over long stretches. So the system gave people a price incentive to use the higher-cost trains and not the lower-cost ones.

 This is the most important thing to resolve about any future fare reductions. Some mechanism is needed to ensure that the most advantageous way to travel between two cities is the one that DB can provide the most efficiently, which is IC/ICE and not RegionalBahn.

 Written by Alon Levy 13 Comments Posted in Germany , Incompetence , Regional Rail , Transportation
 2023-01-25

 Our Construction Costs Report

 This is awkward.

 I was hoping to already release the Transit Costs Project report, but we ran into difficulties after we released and then immediately unreleased; we sent the report to the MTA for feedback, we got feedback, and then we went on a binge of edits. This led to a long cycle of almosts; I thought it would be done at the end of December, then 10 days ago, then tomorrow. The revisions were not huge – there are people who read the original version and the gap between what they’ve seen and what we’re about to release is real but is also far from the difference between complete knowledge (which we don’t have and most likely never will) and complete ignorance.

 We still have minor edits to do, but they’re small, on the order of less than a full day’s work for all three of us, and we can promise that the full report – including the New York report and the synthesis of all reports combined – will be out on February 6th.

 Separately, I’m going to be in New York for about a month after, mostly for non-work reasons. The timing is that there’s an election in Berlin on the 12th, a redo of the city election from 2021 (oddly, not electing for a full term but for the remainder of the term that began in 2021), in which I hope the Greens do better if only so that they’re slightly ahead of SPD and not slightly behind and can replace the scandalized Franziska Giffey as mayor. If I can vote by mail, I’ll be in New York starting a little before the election; if I can’t, I’ll be starting a little later. This will include at least one more in-person event to present our findings and our recommendations for the region and the MTA, at a time to be determined but likely in the second half of February.

 There’s a lot of stuff we didn’t get into the report, because of lack of space; we could have info-dumped and made the synthesis novel-length, but some things had to go (and I will defend every decision to exclude things, and if they turn out to have been necessary, then blame me for being wrong about it). So even if you read the reports you should definitely come to events and ask me specific questions about things that are related to construction costs, such as other projects in the same cities we studied, or issues of the interplay between costs and benefits, or how this applies to urban rail construction going forward.

 Written by Alon Levy Leave a comment Posted in Construction Costs , Personal/Admin , Transportation
 2023-01-24

 Push and Pull Factors and Measuring Modal Shift

 There’s a longstanding debate among activists and academics about what the best way of effecting modal shift from cars to public transport is. Pull factors concern making public transport better through building more rail lines, running them more frequently, improving service convenience, or reducing fares. Push factors concern making driving harder through speed limits, fuel taxes, congestion pricing, and reallocation of street space from cars to public and non-motorized transport. There’s a tendency on the New Left to favor push factors (but the East Asian developmental states are best characterized as push-before-pull and not pure pull).

 This has been refined by researchers at the climate research institute, the Ariadne Project, who published a paper in late 2021 rating various push and pull policies on effectiveness for reducing transport emissions. They conclude that push factors dominate, and pull factors are small, with construction of new public transit almost insignificant, only worth a reduction of around 300,000 tons of CO2 a year Germany-wide, 0.039% of national emissions as of 2021; instituting a 120 km/h speed limit on the Autobahn is said to have about 10 times that effect, while the biggest effects yet would come from carbon taxes. The study laments that pull factors are so much more popular than push factors, which they admit suppress society-wide consumption.

 The research suffers from the same problem as other work in this direction, in that it is bad at estimating the impact of public transport on mode shift. It briefly argues that construction of public transport increases overall consumption and therefore doesn’t do much to reduce emissions. This way, it’s like 2020’s carbon critique of U-Bahn expansion, which I criticized two months ago; the carbon critique argues that each kilometer of U-Bahn built only reduces CO2 emissions by 714 tons a year through mode shift, under the assumption that only 20% of public transport riders are diverted from cars.

 This doesn’t pass a sanity check. 300,000 divided by 714 is 420 km, which is about comparable to the total route length of the four grade-separated U-Bahn systems in Germany plus the Wuppertal Schwebebahn; I think the two figures, 300,000 and 714/km, come from different sources, and judging by the other elements in the study, I suspect 300,000 assumes less construction than a full doubling of Germany’s rapid transit network length. Nonetheless, even under a more generous assumption, this is far too low compared with macro trends in public transport usage.

 The best way to use macro trends as a sanity check is to look at some cases with much more and much less public transport than the present. Do they look like it’s a total difference of 0.039%? No, and that’s even taking into account that transit cities tend to be wealthier, stimulating more consumption and more production. As I pointed out in my post two months ago, while Germany averages 9.15 t-CO2/capita, Berlin only does 5.38, and while Germany averages 580 cars per 1,000 people, Berlin only does 327. The difference is largely about Berlin’s pull factors. Push factors in the city are not extensive, and what exists is implemented only in areas that already have very low car use.

 Even lower household emissions in Berlin must be viewed as downstream of the density that is enabled by the presence of a large urban rail network. Cars are a low-capacity mode of transport, so an auto-oriented region, like American metro regions, has to spread out its homes and destinations to limit congestion, and this increases household emissions (single-family houses emit more than apartment buildings) and also encourages people to travel longer distances for their commute and routine non-commute trips.

 This is not easy to measure. Public transport projects have gotten fairly good in the last generation at estimating ridership, but estimating the responsibility of one particular project to modal shift is hard. It interacts with the entire city region. For example, building one rail line can be measured to shift modes in the neighborhoods it serves, but it also encourages destinations to locate in city center since people from the neighborhoods the line serves can now access it, and the increase in office, retail, and community development then leads to a small modal shift citywide. Worse, trying to tease out the effect of the rail line on modal shift sufficiently carefully may lead researchers to count this citywide effect negatively, since one econometric technique is to compare the neighborhoods near the line with neighborhoods in the same city not on the line.

 In practice, the construction of rail lines tends to co-occur with other policies that improve public transport, which may be pull or push factors. This means that it’s very easy to misattribute the effect of urban rail expansion to those other factors. I am convinced that this is what is happening here; the proper comparison must be at the level of an entire region, looking at the emissions of different regions with different levels of public transport usage.

 The upshot is that if it is hard to measure the effect of public transport construction on modal shift and emissions, then the uncertain factors should not be set to zero. Rather, they should be set to sanity-check levels. For example, one can compare New York with the rest of the United States, since it’s a starker difference between a transit city and an auto-oriented country than anywhere in Europe, and correct for non-transport effects like climate and electricity mix, both of which are easy to measure.

 Within Germany, Berlin has 42% lower emissions than the rest of the country per capita. Berlin achieves this with an urban rail network that, in 2019, got 1,289 million rail trips, nearly all within the city of 3.7 million, a minority in the suburban region of perhaps 1.3 million. This is around 250 trips/person regionwide, and 320/person citywide assigning around 20% of S-Bahn ridership to suburbs like Potsdam and Oranienburg. What’s more, Germany doesn’t start from zero; this is not the United States, with multiple large cities with around 10 annual rail trips per capita. Netting out buses from VDV’s data (p. 25) gets around 6.3 billion rail trips in Germany in 2019 including trams, or 75 per capita.

 The difference between 320 and 75 is around 250 – I know it’s actually 245 but at this point I’m deliberately reducing precision because those are sanity-check estimates and I don’t want people thinking they’re correct to three significant figures (try 1.5). If we attribute the entire Berlin-Germany difference of about 3.8 t-CO2/capita to public transport and downstream changes to the urban layout, then we get 0.015 t saved per annual trip generated. To get from there to 300,000 tons saved, we just need 20 million annual rail riders, or around 65,000 daily ones, which is easy to generate on a single line; the approximately 2 km extension of U8 to Märkisches Viertel that Berlin keeps postponing is estimated to generate 25,000-30,000.

 Now, to sanity-check the sanity check, the estimate here is that every trip on urban rail saves 15 kg-CO2. This is an aggressive figure; new cars nowadays average 100 g/km and averaged 180 g/km in 2001 (source, PDF-p. 15), and the average displaced car trip is not 150 km or even 80 km – Americans average around 45 km/day, or somewhat more when only adults are considered. Rather, the issue is a combination of factors:

 * Because the limiting factor to car transport is capacity, in practice what happens in an auto-oriented region is that it fills from the inside outward, and any modal shift ends up displacing the outermost and longest car trips. I proposed a model for that in a blog post from four years ago.
 * Public transport displaces car trips on a more than one-to-one basis (and certainly more than 20% as in the carbon critique of the U-Bahn). This is because public transport users also walk and bike, and transit cities have high modal splits for active transport by the standards of auto-oriented cities, if not by the standards of Dutch cities. Berlin’s all-trip modal split in 2018 was 26% car, 27% public transport, 18% bike, 30% walking – and the high active transport modal split exists not because of road diets, which are few and far between, but because of the presence of a large core fed by the U- and S-Bahn.
 * Public transport reduces household energy usage by encouraging people to live in apartment buildings with shared walls rather than in single-family houses, which have much greater heating requirements; this is also the mechanism through which transit cities have relatively high usage of active transport even without trying very hard.

 I don’t think these factors fully explain away the gap between 45 km/day and 150 km per trip (so around 300/day), but they explain a large enough fraction of it that the installation of a system like what Berlin has – or, better, what Tokyo has – should be a climate priority. If your model says it doesn’t, it needs a lot more work than to just talk about the consumption effects of more public transport (if you’re bothered by how Berlin is poor for its size, compare New York with the rest of the United States).

 In fact, if estimating modal shift is hard, then it’s best to approximate it by ridership. It’s imperfect because there is the effect of walking and biking; some lines really do just compete with walking, like city-center streetcars, but usually, to first order, it’s a good enough estimate. If it’s hard to estimate the benefits then they should not be set to zero, but rather set proportionally to something easier to measure, in this case ridership. Investment should follow ridership-maximizing strategies, and only deviate from them in corner cases.

 Written by Alon Levy 56 Comments Posted in Environmental Issues , Germany , Good/Interesting Studies , Shoddy Studies , Studies
 2023-01-22

 Paris-Berlin Trains, But no Infrastructure

 Yesterday, Bloomberg reported that Macron and Scholz announced new train service between Paris and Berlin to debut next year, as intercity rail demand in Europe is steadily rising and people want to travel not just within countries but also between them. Currently, there is no direct rail service, and passengers who wish to travel on this city pair have to change trains in Frankfurt or Cologne. There’s just one problem: the train will not have any supportive infrastructure and therefore take the same eight hours that trains take today with a transfer.

 This is especially frustrating, since Germany is already investing in improving its intercity rail. Unfortunately, the investments are halting and partial – right now the longest city pair connected entirely by high-speed rail is Cologne-Frankfurt, a distance of 180 km, and ongoing plans are going to close some low-speed gaps elsewhere in the system but still not create any long-range continuous high-speed rail corridor connecting major cities. With ongoing plans, Cologne-Stuttgart is going to be entirely fast, but not that fast – Frankfurt-Mannheim is supposed to be sped up to 29 minutes over about 75 km.

 Berlin-Paris is a good axis for such investment. This includes the following sections:

 * Berlin-Halle is currently medium-speed, trains taking 1:08-1:16 to do 162 km, but the flat, low-density terrain is easy for high-speed rail, which could speed this up to 40-45 minutes at fairly low cost since no tunnels and little bridging would be required.
 * Halle-Erfurt is already fast, thanks to investments in the Berlin-Munich axis.
 * Erfurt-Frankfurt is currently slow, but there are plans to build high-speed rail from Erfurt to Fulda and thence Hanau. The trip times leave a lot to be desired, but newer 300 km/h trains like the Velaro Novo, and perhaps a commitment to push the line not just to Hanau but closer to Frankfurt itself, could do this section in an hour.
 * Frankfurt-Saarbrücken is very slow. Saarbrücken is at the western margin of Germany and is not significant enough by itself to merit any high-speed rail investment. Between it and Frankfurt, the terrain is rolling and some tunneling is needed, and the only significant intermediate stops are Mainz (close enough to Frankfurt it’s a mere stop of opportunity) and Kaiserslautern. Nonetheless, fast trains could get from Frankfurt to the border in 45 minutes, whereas today they take two hours.

 Unfortunately, they’re not talking about any pan-European infrastructure here. Building things is too difficult, so instead the plan is to run night trains – this despite the fact that Frankfurt-Saarbrücken with a connection to the LGV Est would make a great joint project.

 Written by Alon Levy 45 Comments Posted in Germany , High-Speed Rail , Incompetence , Transportation
 2023-01-19

 Bad Public Transit in the Third World

 There’s sometimes a stereotype that in poor countries with low car ownership, alternatives to the car are flourishing. I saw a post on Mastodon making this premise, and pointed out already in comments that this is not really true. This is a more detailed version of what I said in 500 characters. In short, in most of the third world, non-car transportation is bad, and nearly all ridership (on jitneys and buses) is out of poverty, as is most walking. While car ownership is low, the elites who do own cars dominate local affairs, and therefore cities are car-dominated and not at all walkable, even as 90%+ of the population does not own a car.

 What’s more, the developing countries that do manage to build good public transportation don’t stay developing for long. The same development model of Japan, the East Asian Tigers, and now China has built both rail-oriented cities and high economic growth, to the point that Japan and the Tigers are fully developed, and China is a solidly middle-income economy. The sort of places that stay poor, or get stuck in a middle-income trap, also tend to have stagnant urban rail networks, and so grow more auto-oriented over time.

 The situation in Southeast Asia

 With the exception of Singapore, nowhere in Southeast Asia is public transit good. What’s more, construction costs have been high for elevated lines and very high for underground ones, slowing down the construction of metro systems.

 In Kuala Lumpur and Bangkok, motorization is high and public transit usage is weak. Paul Barter’s thesis details how both cities got this way, in comparison with the more transit-oriented model used in Tokyo, Seoul, Hong Kong, and Singapore. The thesis also predicts that the poorer megacities of Southeast Asia – Jakarta and Manila – will follow the auto-oriented path as they develop, which has indeed happened in the 13 years since it was written.

 The situation in those cities is, to be fair, murky. Manila has a large urban rail under construction right now, with average to above average costs for elevated lines and high ones for subways. But the system it has today consists of four lines, two branded light rail, one branded MRT, and one commuter line. In 2019, the six-month ridership on the system was 162 million. A total of 324 million in a metro area the size of Manila is extraordinarily low: the administrative Metro Manila region has 13.5 million people, and the urban or metropolitan area according to both Citypopulation.de and Demographia is 24-26 million. On the strictest definition of Metro Manila, this is 24 trips per person per year; on the wider ones, it is about 13, similar to San Diego or Portland and only somewhat better than Atlanta.

 Jakarta is in the same situation of flux. It recently opened a half-underground MRT line at fairly high cost, and is modernizing its commuter rail network along Japanese lines, using second-hand Japanese equipment. Commuter rail ridership was 1.2 million a day last year, or around 360 million a year, already higher than before corona; the MRT had 20 million riders last year, and an airport link had 1.5 million in 2018. This isn’t everything – there’s also a short light metro called LRT for which I can’t find numbers – but it wouldn’t be more than second-order. This is 400 million annual rail trips, in a region of 32 million people.

 The future of these cities is larger versions of Bangkok. Thailand is sufficiently middle-income that we can see directly how its transport system evolves as it leaves poverty, and the results are not good. Bus ridership is high, but it’s rapidly falling as anyone who can afford a car gets one; a JICA report about MRT development puts the region’s modal split at 5% MRT, 36% bus, and the rest private (PDF-p. 69) – and the income of bus riders is significantly lower than that of drivers (PDF-p. 229), whereas MRT riders are closer to drivers.

 Even wealthier than Bangkok, with the same auto-oriented system, is Kuala Lumpur. There, the modal split is about 8% bus, 7% train, and the rest private. This is worse than San Francisco and the major cities of Canada and Australia, let alone New York or any large European city. The national modal split in England, France, Germany, and Spain is about 16% – the first three countries’ figures predate corona, but in Spain they’re from 2021, with suppressed public transport ridership. Note that rail ridership per capita is healthier in Kuala Lumpur than in Jakarta or Manila – all rail lines combined are 760,000 riders per day, say 228 million per year, in a region of maybe 7 million people. This is better than a no-transit American city like San Diego, but worse than a bad-transit one like Chicago or Washington, where the modal split is about the same but there is no longer the kind of poverty that is common in Malaysia, let alone in Indonesia, and therefore if people ride the trains it’s because they get them to their city center jobs and not because they’re poor.

 Even in Singapore, the best example out there of a transit-oriented rich city, it took until very recently for MRT coverage to be good enough that people willingly depend on it; it only reached NUS after I graduated. In the 1990s, the epitome of middle-class Singaporean materialism was described as owning the Five Cs, of which one was a car; traffic suppression, a Paul Barter describes, has centered fees on cars, much more car purchase than car use (despite the world-famous congestion pricing system), and thus to those wealthy enough to afford cars, they’re convenient in ways they are not in Paris, Berlin, or Stockholm.

 The situation in Africa

 African countries between the Sahara and the Kalahari are all very poor, with low car ownership. However, they are thoroughly car-dominated.

 From the outside, it’s fascinating to see how the better-off countries in that region, like Nigeria, are already imitating Southeast Asia. Malaysia overregulated its jitneys out of existence because they were messy and this bothered elites, and because it wanted to create an internal market for its state-owned automakers. Nigeria is doing the same, on the former grounds; to the extent it hasn’t happened despite years of trying, it’s because the state is too weak to do more than harass the drivers and users of the system.

 It’s notable that the Lagos discourse about the evils of the danfo – they are noisy, they are polluting, they drive like maniacs – there is little attention to how cars create all the same problems, except at larger scale per passenger served. The local notables drive (or are driven); the people who they scorn as unwashed, overly fecund, criminal masses ride the danfo. Thanks to aggressive domination by cars and inattention to the needs of the non-driving majority, Lagos’s car ownership is high for how poor it is – one source from 2017 says 5 million cars in the state, another from 2021 says 6.5 million vehicles between the state and Kano State. The denominator population in the latter source is 27 million officially, but unofficially likely more; 200 vehicles per 1,000 people is plausible for Lagos, which to be clear is not much less than New York or Paris, on an order of magnitude lower GDP per capita. Tokyo took until about 1970 to reach 100 vehicles per 1,000 people, at which point Japan had almost fully converged with American GDP per capita.

 This is not specific to Lagos. A cousin who spent some time in Kampala told me of the hierarchy on the roads: pedestrians fear motorcycles, motorcycles fear cars, cars fear trucks. There is no pedestrian infrastructure to speak of; a rapid transit system is still a dream, to the point that a crayon proposal that spread on Twitter made local media. That the vast majority of Ugandans don’t own cars doesn’t matter; Kampala remains dominated by the few who do.

 Transit and development

 I don’t think it’s a coincidence that the sort of developing countries that build successful urban rail systems don’t stay poor for long. Part of it is that public transportation is good for economic development, but that’s not most of it – the United States manages to be rich without it except in a handful of cities. Rather, I suspect the reason has to do with state capacity.

 More specifically, the reason cities with 100-200 cars per 1,000 people are thoroughly dominated by cars is that those 10-20% drivers (or people who are driven) are the elites. Their elite status can come from any source – passive business income, landlordism, active business income, skilled professional work – but usually it tilts toward the traditional, i.e. passive. These groups tend to be incredibly anti-developmental: they own small businesses, sometimes actively and sometimes passively, and resent being made redundant through economies of scale. India has problems with economic dwarfism and informality, and this is typical of poor countries; if anything, India is better than most at developing a handful of big businesses in high-value added industries.

 The upshot is that the sort of people who drive, and especially the sort of drivers who are powerful enough to effect local changes to get incremental upgrades to roads at the expense of non-drivers, are usually anti-developmental classes. The East Asian developmental states (and Singapore and Hong Kong, which share many characteristics with them) clamped down on such classes hard, on either nationalist or socialist grounds; Japan, both Koreas, and both Chinas engaged in land reform, with characteristic violence in the two socialist states and without it but still with forcible purchase in the three capitalist states. The same sort of state that can eliminate landlordism can also, as a matter of capital formation, clamp down on consumption and encourage personal savings, producing atypically low levels of motorization well into middle-income status. Singapore, whose elite consumption centers vacations out of the country, has managed to do so even as a high-income country – and even more normal Tokyo and Seoul have much higher rail usage and lower car usage than their closest Western analog, New York.

 India is in many ways anti-developmental, but it does manage to grow. Its anti-developmentalism is anti-urban and NIMBY, but it is capable of building infrastructure. Its metro program has problems with high construction costs (but Southeast Asia’s are generally worse) and lack of integration with other modes such as commuter rail, which the middle class denigrates as only befitting poor people; but the Delhi Metro had 5.5 million daily riders just before corona, slightly behind New York in a slightly larger metro area, perhaps a better comparison than Jakarta and Manila’s San Diego.

 It’s the slower-growing developing countries that are not managing to even build the systems India has, let alone East Asia. They don’t have high car use, but only because they are poor, and in practice, they are thoroughly car-dominated, and everyone who doesn’t have a car wants one. A rich country really is not one where even the poor have cars but where even the rich use public transportation – and those countries aren’t rich and don’t grow at rates that will make them rich.

 Written by Alon Levy 100 Comments Posted in Incompetence , Transportation , Urban Transit
 2023-01-12

 High Costs are not About Precarity

 I’ve seen people who I think highly of argue that high construction costs in the United States are an artifact of precarity. The argument goes that the political support for public transportation there is so flimsy that agencies are forced to buy political support by spending more money than they need. This may include giving in to NIMBY pressure to use costlier but less impactful (or apparently less impactful) techniques, to spread money around with other government agencies and avoid fighting back, to build extravagant and fancier-looking but less standardized stations, and so on. The solution, per this theory, is to politically support public transportation construction more so that transit agencies will have more backing.

 This argument also happens to be completely false, and the solution suggested is counterproductive. In fact, the worst cost blowouts are for the politically most certain projects; Second Avenue Subway enjoyed unanimous support in New York politics.

 Cost-effectiveness under precarity

 Three projects relevant to our work at the Transit Costs Project have been done exceptionally cost-effectively in an environment of political uncertainty: the T-bana, the LGV Sud-Est, and Bahn 2000.

 T-bana

 The original construction of the T-bana was done at exceptionally low cost. We go over this in the Sweden report to some extent, but, in short, between the 1950s and 70s, the total cost of the system’s construction was 5 billion kronor in 1975 prices, which built around 100 km, of which 57% are underground. In PPP 2022 dollars, this is $3.6 billion, or $35 million/km, not entirely but mostly underground. This was low for the time: for example, in London, the Victoria line was $122 million/km and the Jubilee line was $172 million/km (source, p. 78), and Italian costs in the 1960s and 70s were similar, averaging $129 million/km before 1970.

 The era of Social Democrat dominance in Swedish politics on hindsight looks like one of consensus in favor of big public projects. But the T-bana itself was controversial. When the decision was made to build it in the 1940s, Stockholm County had about 1 million people; at the time, metros were present in much larger cities, like New York, London, Paris, Berlin, and Tokyo, and it was uncertain that a city the size of Stockholm would need such a system. Its closest analog, Copenhagen, did not build such a system until the 1990s, when it was a metro region of 2 million. It was uncertain that Stockholm should need rapid transit, and there were arguments for and against it in the city. Nor was there any transit-first policy in postwar Sweden: urban planning was the same modernist combination of urban renewal, automobile scale, and tower-in-a-park housing, and outside Stockholm County, the Million Program projects were thoroughly car-oriented.

 Construction costs in Sweden are a lot higher now than they were in the 1950s, 60s, and 70s. Nya Tunnelbanan is $230 million/km, compared with a post-1990s Italian average of $220 million; British costs have exploded in tandem, so that now the Underground extensions clock at $600 million/km. Our best explanation is that the UK adopted what we call the globalized system of procurement, privatizing planning functions to consultants and privatizing risk to contractors, which creates more conflict; the UK also has an unusually high soft cost factor. From American data (and not just New York) and some British data, I believe that the roughly 2.5 cost premium of the UK over Italy is entirely reducible to such soft costs, procurement conflict, risk compensation, and excessive contingency. And yet, Sweden itself, with some elements of the same globalized system, maintains a roughly Italian cost level, albeit trending the wrong way.

 And today, too, the politics of rail expansion in Sweden are uncertain. There was controversy over both Citybanan and Nya Tunnelbanan, neither of which passed a cost-benefit analysis (for reasons that I believe impugn the cost-benefit analysis more than those projects); it was uncertain that either would be funded. Controversy remained over plans to build high-speed rail connecting Stockholm with Gothenburg and Malmö, and the newly-elected right-wing government just canceled them in order to prioritize investment in roads. Swedish rail projects today remain precarious, and have to justify themselves on cost and efficiency grounds.

 LGV Sud-Est

 Like nearly all other rich countries, France was hit hard by the 1973 oil crisis; economic growth there and in the US, Japan, and most of the rest of Western Europe would never be as high as it was between the end of WW2 and the 1970s (“Trente Glorieuses“). On hindsight, France’s response to the crisis models can-go governance, with an energy saving ad declaring “in France we don’t have oil, but we have ideas.” The French state built nuclear power plants with gusto, peaking around 90% of national electricity use – and even today’s reduced share, around 70%, is by a large margin the highest in the world. At the same time, it built a high-speed rail network, connecting Paris with most other provincial cities at some of the highest average speeds outside China between major cities, reaching about 230 km/h between Paris and Marseille and 245 km/h between Paris and Bordeaux; usage per capita is one of the highest in Europe and, measured in passenger-km, not too bad by East Asian standards.

 But in fact, the first LGV, the LGV Sud-Est, was deeply controversial. At the time, the only high-speed rail network in operation was the Shinkansen, and while France learned more from Japan than any other European country (for example, the RER was influenced by Tokyo rail operations), the circumstances for intercity were completely different. SNCF had benefited from having done many of its own experiments with high-speed technology, but the business case was murky. SNCF had to innovate in running an open system, with extensive through-running to cities off the line, which Japan would only introduce in the 1990s with the Mini-Shinkansen.

 Within the French state, the project was controversial. Anthony Perl’s New Departures details how there were people within the government who wanted to cancel it entirely as it was unaffordable. At the end, the French state didn’t finance the line, and required SNCF to find private loans on the international market, though it did guarantee those loans. It also delayed the line’s opening: instead of opening the entire line from Paris to Lyon in one go, it opened two-thirds of it on the Lyon side in 1981 and the last third into Paris in 1983, requiring trains to run on the classical line at low speed between Paris and Saint-Florentin for two years; in that era, phased opening was uncommon, and lines generally opened to the end at once, such as between Tokyo and Shin-Osaka.

 Construction was extraordinarily inexpensive. In PPP 2022 dollars, it cost $8.4 million/km. This is, by a margin, the lowest-cost high-speed rail line ever built that I know about. The Tokaido Shinkansen cost 380 billion yen, or in PPP 2022 dollars $40 million/km, representing a factor of two cost overrun that forced JNR’s head to resign. Spain has unusually low construction costs, and even there, Madrid-Seville was $15.7 million/km. SNCF innovated in every way possible to save money. Realizing that high-speed trains could climb steeper grades, it built the LGV Sud-Est with a ruling grade of 3.5%, which has since become a norm in and around Europe, compared with the Shinkansen’s 1.5-2%; the line has no tunnels, unlike the classical Paris-Lyon line. It built the line on the ground rather than on viaducts, and balanced cut and fill locally so that material cut to grade the line could be used for nearby fill. Thanks to the line’s low costs and high ridership, the financial return on investment for SNCF has been 15%, and social return on investment has been 30% (source, pp. 11-12).

 This cost-effectiveness would never recur. The line’s success ensured that LGV construction would enjoy total political backing. The core features of LGV construction are still there – earthworks rather than viaducts, 3.5% grades, limited tunneling, overcompensation of landowners by about 30% with land swap deals to defuse the possibility of farmer riots. But the next few lines cost about $20 million/km or slightly less, and this cost has since crept up to about $30 million/km or even more. This remains low by international standards (but not by Spanish ones), but the trend is negative.

 SNCF is coasting on its success from a generation ago, secure that funding for LGVs and state support in political contention is forthcoming, and the routing decisions have grown worse. In response to NIMBYism in Provence, the French state assented to a tunnel-heavy route, including a conversion of Marseille from an at-grade terminal to an underground through-station, akin to Stuttgart 21, which has not been done before in France, and the resulting high costs have led to delays on the project. Operations have grown ever more airline-style, experimenting with low-cost airline imitation to the point of reducing fare receipts without any increase in ridership. One of the French consultants we’ve spoken with said that their company’s third-party design costs are 7-8% of the hard costs, which figure is similar to what we’ve seen in Italy and to the in-house rate in Spain – but the same consultant told us that there is so much bloat at SNCF that when it designs its own projects, the costs are not 7% but 25%, a figure in line with American rates.

 Bahn 2000

 Switzerland has Europe’s strongest passenger rail network by all measures: highest traffic measured by passenger-km per capita, highest modal split for passenger-km, highest traffic density. Its success is well-known in surrounding countries, which are gradually either imitate its methods or, in the case of Germany, pretending to do so. It has achieved its success through continuous improvement over the generations, but the most notable element of this system was implemented in the 1990s as part of the Bahn 2000 project.

 The current system is based on a national-scale clockface system (“Takt”) with trains repeating hourly, with the strongest links, like the Zurich-Bern-Basel triangle, running every half hour. Connections are timed in those three cities and several others, called knots, so that trains enter each station a few minutes before the connection time (usually the hour) and depart a few minutes after, permitting passengers to get between most pairs of Swiss cities with short transfers. Reliability is high, thanks to targeted investments designed to ensure that trains could make those connections in practice and not just in theory. Further planning centers adding more knots and expanding this system to the periphery of Switzerland.

 Switzerland is famous for its consensus governance system – its plural executive is drawn from the four largest parties in proportion to their votes, with no coalition vs. opposition politics. But the process that led to the decision to adopt Bahn 2000 was not at all one of unanimity. There had been plans to build high-speed rail, as there were nearly everywhere else in Western Europe. But they were criticized for their high costs, and there was extensive center-right pressure to cut the budget. Bahn 2000 was thus conceived in an environment of austerity. Many of its features were explicitly about saving money:

 * The knot system is connected with running trains as fast as necessary, not as fast as possible. Investments in speed are pursued only insofar as they permit trains to make their connections; higher speeds are considered gratuitous.
 * Bilevel trains are an alternative to lengthening the platforms.
 * Timed overtakes and meets are an alternative to more extensive multi-tracking of lines.
 * Investment in better timetabling and systems (the electronics side of the electronics-before-concrete slogan) is cheaper than adding tunnels and viaducts.

 Swiss megaprojects have to go to referendum, and sometimes the referendums return a no; this happened with the Zurich U-Bahn twice, leading to the construction of the S-Bahn instead. All Swiss planners know in a country this small and this fiscally conservative, any extravagance will lead to rejection. The result is that they’ve instead optimized construction at all levels, and even their unit costs of tunneling are low; thanks to such optimization, Switzerland has been able to build a fairly extensive medium-speed rail system, with more tunneling per capita than Germany (let alone France), and with two S-Bahn trunk tunnels in Zurich, where no German city today has more than one.

 The American situation

 The worst offenders in the United States are not at all politically precarious. There is practically unanimous consensus in New York about the necessity of Second Avenue Subway. At no point was the project under any threat. There is an ideological right in the city, rooted less in party politics and more in the New York Post and the Manhattan Institute, with a law-and-order agenda and hostility to unions and to large government programs, but at no point did they call for cancellation; the Manhattan Institute’s Nicole Gelinas has proposed pension cuts for workers and rule changes reducing certain benefits, but not canceling Second Avenue Subway.

 At intercity scale, the same is true of Northeast Corridor investment. The libertarian and conservative pundits who say passenger rail is a waste of money tend to except the Northeast Corridor, or at least its southern half. When the Republicans won the 2010 midterm election, the new chair of the House of Representatives Transportation Committee, John Mica (R-FL), proposed a bill to seek private concessionaires to run intercity rail on the corridor. He did not propose canceling train service, even though in the wake of the same election, multiple conservative governors canceled intercity rail investments in their state, both high-speed (Florida) and low-speed (Wisconsin, Ohio).

 In fact, both programs – New York subway expansion and the Northeast Corridor – are characterized by continuity across partisan shifts, as in more established consensus governance systems. The Northeast Corridor is especially notable for how little role ideological or partisan politics has played so far. New York has micromanagement by politicians – Andrew Cuomo had his pet projects in Penn Station Access and the backward-facing LaGuardia air train, and now Kathy Hochul has hers in the Interborough Express – but Second Avenue Subway was internal, and besides, political micromanagement is a different problem from political precarity.

 And neither of these programs has engaged in any cost control. To the contrary, both are run as if money is infinite. The MTA would surrender to NIMBYs (“good neighbor policy”) and to city agencies looking to extract money from it. It built oversize stations. It spent money protecting buildings from excessive settlement that have been subsequently demolished for redevelopment at higher density.

 The various agencies involved in the Northeast Corridor, likewise, are profligate, and not for lack of political support. Connecticut is full of NIMBYs; one of the consultants working on the plan a few years ago told me there was informal pressure not to ruffle feathers and not to touch anything in the wealthiest suburbs in the state, those of Fairfield County. In fact, high-speed rail construction would require significant house demolitions in the state’s second wealthiest town, Darien – but Darien is so infamously exclusive (“Darien rhymes with Aryan,” say other suburbanites in the area) that the rest of the region feels little solidarity with it.

 NIMBYs aside, there has not been any effort at coordinating the different agencies in the Northeast along anything resembling the Swiss Takt system. This is not about precarity, because this is not a precarious project; this is about total ignorance and incuriosity about best practices, which emanate from a place that doesn’t natively speak English and doesn’t trade in American political references.

 The Green Line Extension

 Boston’s GLX is a fascinating example of cost blowouts without precarity. The history of the project is that its first iteration was pushed by Governor Deval Patrick (D), with the support of groups that sued the state for its delays in planning the project in the 1990s, as a court-mandated mitigation for the extra car traffic induced by the construction of the Big Dig. Patrick instituted a good neighbor policy, in which everything a community group wanted, it would get. Thus, stations were to become neighborhood signatures, and the project was laden with unrelated investments, called betterments, like a $100 million 3 km bike lane called the Somerville Community Path.

 At no point in the eight years Patrick was in power was there a political threat to the project. It was court-mandated, and the extractive local groups that live off of suing the government favored it. The Obama administration was generous with federal stimulus funding, and the designs were rushed in order to use stimulus funding to pay for the project’s design, which would be done by consultants rather than with an expansion of in-house hiring. It’s in this atmosphere of profligacy that the project’s cost exploded to, in today’s money, around $3.5 billion, for 7 km of light rail in existing rights-of-way (albeit ones requiring overpass reconstruction).

 The project did fall under political threat, after Charlie Baker (R) won the 2014 election. Baker’s impact on Massachusetts governance is fascinating, in that he unambiguously cut its budget significantly in the short run, but also had both before (as budget director in the 1990s) and during (through his actions as governor) wrecked the state’s long-term ability to execute infrastructure, setting up a machine intended to privatize the state and avoiding any in-house hiring. Nonetheless, the direct impact of precarity on GLX was to reduce scope: the betterments were removed and the stations were changed from too big to too small. The final cost was $2.3 billion, or around $2.8 billion in 2022 dollars, and half of that was sunk cot from the previous iteration.

 There was no expectation that the project would be canceled – indeed, it was not. A Republican victory was unexpected in a state this left-wing. Then, as Baker was taking office, past governors from both parties expressed optimism that he would get not just GLX done but also the much more complex North-South Rail Link tunnel. Nor did contractors have their contracts yanked unexpectedly, which would get them to bid higher for risk compensation. Baker cold-shouldered not jut NSRL but also much smaller investments in commuter rail, but at no point was anyone stiffed in the process. No: the Patrick-era project was just poorly managed.

 Project selection

 As one caveat, I want to point out a place where precarity does lead to poor project cost-effectiveness: the project selection stage, in the context of tax referendums, especially in Southern California. None of this leads to high per-kilometer costs – Los Angeles’s are exactly as bad as in the rest of the United States – but it does lead to poor project selection, as an unintended consequence of anti-tax New Right politics.

 The California-specific issue is that raising taxes requires a referendum, with a two-thirds vote. Swiss referendums are by simple majority, or at most double majority – but in practice most referendums that have a popular vote majority, even a small one, also have a double majority when needed, and in the last 15 years I believe the only two times they didn’t had the double majority veto overturning a 1.5% margin and a 9% margin.

 California’s requirement of a two-thirds majority was intended to stop wasteful spending and taxation, but has had the opposite effect. In and around San Francisco, the voters are sufficiently left-wing that two-thirds majorities for social policy are not hard to obtain. But in Southern California, they are not; to build public transportation infrastructure, Los Angeles and San Diego County governments cannot rely on ideology, but instead cut deals with local, non-ideological actors through promising them a piece of the package. Los Angeles measures for transit expansion are, by share of money committed, largely not about transit expansion but operations, new buses, or leakage to roads and bridges; then, what does go to transit expansion is divvied by region, with each region getting something, no matter how cost-ineffective, while core improvements are neglected and so are cross-regional connections (since the local extractive actors aren’t going to ride the trains and can’t tell a circumferential project is useful for them).

 This is not a US-wide phenomenon. It’s not even California-wide: this problem is absent from the Bay Area, where BART decided against an expansion to Livermore that was unpopular among technically-oriented advocates, and would like to build more core capacity if it could do so for much less than a billion dollars per km. New York does not have it at all (for one, it doesn’t require two-thirds majorities for budgeting).

 At intercity scale, this precarity does cause Amtrak to maximize how many states and congressional districts it runs money-losing daily trains in. But wasting money on night trains and on peripheral regions is hardly a US-only problem – Japan National Railways did that until well past privatization, when its successors spun off money-losing branch lines to prefecture-subsidized third sector railways. This is not at all why there is no plan for Northeastern intercity rail that is worth its weight in dirt: Northeastern rail improvements have been amply funded relative to objective need (if not relative to American costs), and solid investments in the core coexist with wastes of money on the periphery of the network in many countries.

 The issue of politicization

 The precarious, low-cost examples all had to cut costs because of fiscal pressure. However, in all three, the pressure did not include any politicization of engineering questions. Sweden was setting up a civil service modeled on the American Bureau of Public Roads, currently the Federal Highway Administration, which in the middle of the 20th century was a model of depoliticized governance. France and Switzerland have strong civil service bureaucracies – if anything SNCF is too self-contained and needs reorganization, just not if it’s led by the usual French elites or by people from the airline industry.

 Importantly, low-cost countries with more clientelism and politicization of the state tend to be more deferential to the expertise of engineers. Greece has a far worse problem of overreliance on political appointees than the United States, let alone the other European democracies; but engineering is somewhat of an exception. Hispanic and Portuguese-speaking cultures put great prestige on engineering, reducing the extent of political micromanagement, even in countries without strong apolitical civil service bureaucracies. Even in Turkey, the politicization of public transportation is entirely at macro level: AKP promises to prioritize investment in areas that vote for it and has denied financing to the Istanbul Metro since the city flipped to the opposition (the city instead borrows money from the European Investment Bank), but below that level there is no micromanagement.

 The American examples, in contrast, show much more political micromanagement. This is part of the same package as the privatization of state planning in the globalized system; in the United States often there was never the depoliticization that most of the rest of the developed and middle-income world had, but on top of that, the tendency has been to shut down in-house planning departments or radically shrink them and replace them with consultants. The consultants are then supervised by political appointees with no real qualifications to head capital programs, and the remaining civil servants are browbeaten not to disagree with the political appointees’ proclamations.

 Those political appointees rarely measure themselves by any criteria of infrastructure utility. Even in New York they and the managers don’t consistently use the system; in Los Angeles, they use it about as often as the executive director of a well-endowed charity eats at a soup kitchen. To them, the cost is itself a measure of success – and this is true of other agencies, which treat obtaining other people’s money as a mark of achievement and as testament to their power. This behavior then cascades to local advocacy groups, which try to push solutions that maximize outside funding and are at bet indifferent and at worst actively hostile to any attempt at efficiency.

 Just giving more support to agencies in their current configuration is not going to help. To the contrary, it only confirms that profligacy gets rewarded. A program of depoliticization of the state is required in tandem with expanding in-house hiring and reversing the globalized system, and the political appointees and the managers and political advocates who are used to dealing with them don’t belong in this program.

 Written by Alon Levy 55 Comments Posted in Construction Costs , New York , Sweden , Transportation
 2023-01-07

 Edge Cities With and Without Historic Cores

 An edge city is a dense, auto-oriented job center arising from nearby suburban areas, usually without top-down planning. The office parks of Silicon Valley are one such example: the area had a surplus of land and gradually became the core of the American tech industry. In American urbanism, Tysons in Virginia is a common archetype: the area was a minor crossroads until the Capital Beltway made it unusually accessible by car, providing extensive auto-oriented density with little historic core.

 But there’s a peculiarity, I think mainly in the suburbs of New York. Unlike archetypal edge cities like Silicon Valley, Tysons, Century City in Los Angeles, or Route 128 north of Boston, some of the edge cities of New York are based on historic cores. Those include White Plains and Stamford, which have had booms in high-end jobs in the last 50 years due to job sprawl, but also Mineola, Tarrytown, and even New Brunswick and Morristown.

 The upshot is that it’s much easier to connect these edge cities to public transportation than is typical. In Boston, I’ve spent a lot of time trying to figure out good last mile connections from commuter rail stations. Getting buses to connect outlying residential areas and shopping centers to town center stations is not too hard, but then Route 128 is completely unviable without some major redesign of its road network: the office parks front the freeway in a way that makes it impossible to run buses except dedicated shuttles from one office park to the station, which could never be frequent enough for all-day service. Tysons is investing enormous effort in sprawl repair, which only works because the Washington Metro could be extended there with multiple stations. Far and away, these edge cities are the most difficult case for transit revival for major employment centers.

 And in New York, because so much edge city activity is close to historic cores, this is far easier. Stamford and White Plains already have nontrivial if very small transit usage among their workers, usually reverse-commuters who live in New York and take Metro-North. Mineola could too if the LIRR ran reverse-peak service, but it’s about to start doing so. Tarrytown and Sleepy Hollow could be transit-accessible. The New Jersey edge cities are harder – Edison and Woodbridge have lower job density than Downtown Stamford and Downtown White Plains – but there are some office parks that could be made walkable from the train stations.

 I don’t know what the history of this peculiar feature is. White Plains and Mineola are both county seats and accreted jobs based on their status as early urban centers in regions that boomed with suburban sprawl in the middle of the 20th century. Tarrytown happened to be the landfall of the Tappan Zee Bridge. Perhaps this is what let them develop into edge cities even while having a much older urban history than Tysons (a decidedly non-urban crossroads until the Beltway was built), Route 128, or Silicon Valley (where San Jose was a latecomer to the tech industry).

 What’s true is that all of these edge cities, while fairly close to train stations, are auto-oriented. They’re transit-adjacent but not transit-oriented, in the following ways:

 * The high-rise office buildings are within walking distance to the train station, but not with a neat density gradient in which the highest development intensity is nearest the station.
 * The land use at the stations is parking garages for the use of commuters who drive to the station and use the train as a shuttle from a parking lot to Manhattan, rather than as public transportation the way subway riders do.
 * The streets are fairly hostile to pedestrians, featuring fast car traffic and difficult crossing, without any of the walkability features that city centers have developed in the last 50 years.

 The street changes required are fairly subtle. Let us compare White Plains with Metrotown, both image grabs taken from the same altitude:

 These are both edge cities featuring a train station, big buildings, and wide roads. But in Metrotown, the big buildings are next to the train station, and the flat-looking building to its north is the third-largest shopping mall in Canada. The parking goes behind the buildings, with some lots adjoining Kingsway, which has a frequent trolleybus (line 19) but is secondary as a transportation artery to SkyTrain. Farther away, the residential density remains high, with many high-rises in the typical thin-and-tall style of Vancouver. In contrast, in White Plains, one side of the station is a freeway with low-density residential development behind it, and the other is parking garages with office buildings behind them instead of the reverse.

 The work required to fix this situation is not extensive. Parking must be removed and replaced with tall buildings, which can be commercial or residential depending on demand. This can be done as part of a transit-first strategy at the municipal level, but can also be compelled top-down if the city objects, since the MTA (and other Northeastern state agencies) has preemption power over local zoning on land it owns, including parking lots and garages.

 On the transit side, the usual reforms for improvements in suburban trains and buses would automatically make this viable: high local frequency, integrated bus-rail timetables (to replace the lost parking), integrated fares, etc. The primary target for such reforms is completely different – it’s urban and inner-suburban rail riders – but the beauty of the S-Bahn or RER concept is that it scales well for extending the same high quality of service to the suburbs.

 Written by Alon Levy 36 Comments Posted in New York , Regional Rail , Transportation , Urban Design , Urbanism
 2022-12-30

 The Four Quadrants of Cities for Transit Revival

 Cities that wish to improve their public transportation access and usage are in a bind. Unless they’re already very transit-oriented, they have not only an entrenched economic elite that drives (for example, small business owners almost universally drive), but also have a physical layout that isn’t easy to retrofit even if there is political consensus for modal shift. Thus, to shift travel away from cars, new interventions are needed. Here, there is a distinction between old and new cities. Old cities usually have cores that can be made transit-oriented relatively easily; new cities have demand for new growth, which can be channeled into transit-oriented development. Thus, usually, in both kinds of cities, a considerably degree of modal shift is in fact possible.

 However, it’s perhaps best to treat the features of old and new cities separately. The features of old cities that make transit revival possible, that is the presence of a historic core, and those of new cities, that is demand for future growth, are not in perfect negative correlation. In fact, I’m not sure they consistently have negative correlation at all. So this is really a two-by-two diagram, producing four quadrants of potential transit cities.

 Old cities

 The history of public transportation is one of decline in the second half of the 20th century in places that were already rich then; newly-industrialized countries often have different histories. The upshot is that an old auto-oriented place must have been a sizable city before the decline of mass transit, giving it a large core to work from. This core is typically fairly walkable and dense, so transit revival would start from there.

 The most successful examples I know of involve the restoration of historic railroads as modern regional lines. Germany is full of small towns that have done so; Hans-Joachim Zierke has some examples of low-cost restoration of regional lines. Overall, Germany writ large must be viewed as such an example: while German economic growth is healthy, population growth is anemic, and the gradual increase in the modal split for public transportation here must be viewed as more intensive reuse of a historic national rail network, anchored by tens of small city cores.

 At the level of a metropolitan area, the best candidates for such a revival are similarly old places; in North America, the best I can think of for this are Philadelphia, Boston, and Chicago. Americans don’t perceive any of the three as especially auto-oriented, but their modal splits are comparable to those of small French cities. But in a way, they show one way forward. If there’s a walkable, transit-oriented core, then it may be attractive for people to live near city center; in those three cities it’s also possible to live farther away and commute by subway, but in smaller ones (say, smaller New England cities), the subway is not available but conversely it’s usually affordable to live within walking distance of the historic city center. This creates a New Left-flavored transit revival in that it begins with the dense city center as a locus of consumption, and only then, as a critical mass of people lives there, as a place that it’s worth building new urban rail to.

 New cities

 Usually, if a city has a lot of recent growth from the era in which it has become taken for granted that mobility is by car, then it should have demand for further growth in the future. This demand can be planned around growth zones with a combination of higher residential density and higher job density near rail corridors. The best time to do transit-oriented development is before auto-oriented development patterns even set in.

 There are multiple North American examples of how this works. The best is Vancouver, a metropolitan area that has gone from 560,000 people in the 1951 census to 2.6 million in the 2021 census. Ordinarily, one should expect such a region to be entirely auto-oriented, as most American cities with almost entirely postwar growth are; but in 2016, the last census before corona, it had a 20% work trip modal split, and that was before the Evergreen extension opened.

 Vancouver has achieved this by using its strong demand for growth to build a high-rise city center, with office towers in the very center and residential ones ringing it, as well as high-density residential neighborhoods next to the Expo Line stations. The biggest suburbs of Vancouver have followed the same plan: Burnaby built an entirely new city center at Metrotown in conjunction with the Expo Line, and even more auto-oriented Surrey has built up Whalley, at the current outer terminal of the line, as one of its main city centers. Housing growth in the region is rapid; YIMBY advocacy calls for more, but the main focus isn’t on broad development (since this already happens) but on permitting more housing in recalcitrant rich areas, led by the West Side, which will soon have its Broadway extension of the Millennium Line.

 Less certain but still interesting examples of the same principle are Calgary, Seattle, and Washington. Calgary, a low-density city, planned its growth around the C-Train, and built a high-rise city center, limiting job sprawl even as residential sprawl is extensive; Seattle and the Virginia-side suburbs of Washington have permitted extensive infill housing and this has helped their urban rail systems achieve high ridership by American standards, Seattle even overtaking Philadelphia’s modal split.

 The four quadrants

 The above contrast of old and new cities misses cities that have positive features of both – or neither. The cities with both positive features have the easiest time improving their public transportation systems, and many have never been truly auto-oriented, such as New York or Berlin, to the point that they’re not the best examples to use for how a more auto-oriented city can redevelop as a transit city.

 In North America, the best example of both is San Francisco, which simultaneously is an old city with a high-density core and a place with immense demand for growth fueled by the tech industry. The third-generation tech firms – those founded from the mid-2000s onward (Facebook is in a way the last second-generation firm, which generation began with Apple and Microsoft) – have generally headquartered in the city and not in Silicon Valley. Twitter, Uber, Lyft, Airbnb, Dropbox, and Slack are all in the city, and the traditional central business district has expanded to South of Market to accommodate. This is really a combination of the consumption-oriented old-city model, as growing numbers of employees of older second-generation firms chose to live in the city and reverse-commute to Silicon Valley, and the growth-oriented new-city model. Not for nothing, the narrower metropolitan statistical area of San Francisco (without Silicon Valley) reached a modal split of 17% just before corona, the second highest in the United States, with healthy projections for growth.

 But then there is the other quadrant, comprising cities that have neither the positive features of old cities nor those of new cities. To be in this quadrant, a city must not be so old as to have a large historic core or an extensive legacy rail network that can be revived, but also be too poor and stagnant to generate new growth demand. Such a city therefore must have grown in a fairly narrow period of time in the early- to mid-20th century. The best example I can think of is Detroit. The consumption-centric model of old city growth can work even there, but it can’t scale well, since there’s not enough of a core compared with the current extent of the population to build out of.

 Written by Alon Levy 78 Comments Posted in Development , Germany , Urban Design , Urbanism , Vancouver
 Older Posts
 Search for:

 Recent Posts

 * FDP and Vice Signaling
 * Quick Note: High Speed 2 and Euston
 * Free Public Transport, Fare Integration, and Capacity
 * Our Construction Costs Report
 * Push and Pull Factors and Measuring Modal Shift

 Recent Comments

 Henry Miller on FDP and Vice Signaling 
 A má qualidade do tr… on Growth Without Urbanization
 Frederick on Quick Note: High Speed 2 and… 
 gdlhoybus on Large-Diameter TBMs 
 gdlhoybus on Quick Note: High Speed 2 and… 
 adirondacker12800 on FDP and Vice Signaling 
 Matthew Hutton on FDP and Vice Signaling 
 Jason on FDP and Vice Signaling 
 Jason on Free Public Transport, Fare In… 
 Sascha Claus on FDP and Vice Signaling 
 Sascha Claus on FDP and Vice Signaling 
 Sascha Claus on FDP and Vice Signaling 
 Borners on FDP and Vice Signaling 
 Reedman Bassoon on FDP and Vice Signaling 
 Alon Levy on FDP and Vice Signaling 
 

 Blogroll

 * Aaron Renn (the Urbanophile)
 * Alexander Rapp
 * Auckland Transport Blog
 * Better Institutions
 * Burning the Midnight Oil for Energy Independence
 * California HSR Blog
 * Caltrain-HSR Compatibility Blog
 * Cap'n Transit Rides Again
 * City Notes (Daniel Kay Hertz)
 * Corner Side Yard
 * Crossing the Lines
 * Fbfree
 * Gateway Streets
 * Greater City: Providence
 * Greater Greater Washington
 * Green Caltrain
 * Human Transit
 * Humanity in Transit
 * Itinerant Urbanist
 * Larry Littlefield
 * Let's Go LA
 * Lewyn Addresses America
 * London Reconnections
 * M1EK's Bake Sale of Bile
 * Market Urbanism
 * Old Urbanist
 * Pennsylvania HSR
 * Pittsburgh, Youngstown, Akron, Cleveland Arts And Livable City Blog
 * Public Transport at About.com
 * Queens Transit
 * Reason and Rail
 * Reinventing Transport
 * Richard Mlynarik
 * Second Avenue Sagas
 * Stop and Move
 * Streetsblog Network
 * Strong Towns
 * Systemic Failure
 * The Austin Contrarian
 * The Black Urbanist
 * The LIRR Today
 * The Overhead Wire
 * The Transport Politic
 * Train Star
 * Transit Futures

 Places I write for

 * Baltimore Brew
 * Bay City Beacon
 * City Metric
 * CityLab
 * Curbed
 * DC Policy Center
 * Metro Report
 * New York YIMBY
 * PlanPhilly
 * Railway Gazette
 * Streetsblog Network
 * The American Interest
 * Urbanize.LA
 * Voice of San Diego
 * Vox

 Supporters

 * Aaron Renn (the Urbanophile)
 * Alexander Rapp
 * Andre Lot, research scholar in finance
 * Andrew Obus
 * Bruce Steinberg
 * D. W. Rowlands
 * Dr2chase
 * Fbfree
 * Gillian Gillett
 * https://buyer.bike/
 * Jim Aloisi, Trimount Consulting and Board Member at TransitMatters
 * Lawrence Velázquez
 * Let's Go LA
 * Lewis Lehe
 * Manuela Zoninsein
 * Matthew Danish
 * Matthew Frank
 * Michael Druker
 * Nicholas Ober
 * Peter Furth
 * Peter Gault
 * Varpilah K Chase

 Archives

 * January 2023
 * December 2022
 * November 2022
 * October 2022
 * September 2022
 * August 2022
 * July 2022
 * June 2022
 * May 2022
 * April 2022
 * March 2022
 * February 2022
 * January 2022
 * December 2021
 * November 2021
 * October 2021
 * September 2021
 * August 2021
 * July 2021
 * June 2021
 * May 2021
 * April 2021
 * March 2021
 * February 2021
 * January 2021
 * December 2020
 * November 2020
 * October 2020
 * September 2020
 * August 2020
 * July 2020
 * June 2020
 * May 2020
 * April 2020
 * March 2020
 * February 2020
 * January 2020
 * December 2019
 * November 2019
 * October 2019
 * September 2019
 * August 2019
 * July 2019
 * June 2019
 * May 2019
 * April 2019
 * March 2019
 * February 2019
 * January 2019
 * December 2018
 * November 2018
 * October 2018
 * September 2018
 * August 2018
 * July 2018
 * June 2018
 * May 2018
 * April 2018
 * March 2018
 * February 2018
 * January 2018
 * December 2017
 * November 2017
 * October 2017
 * September 2017
 * August 2017
 * July 2017
 * June 2017
 * May 2017
 * April 2017
 * March 2017
 * February 2017
 * January 2017
 * December 2016
 * November 2016
 * October 2016
 * September 2016
 * August 2016
 * July 2016
 * June 2016
 * May 2016
 * April 2016
 * March 2016
 * February 2016
 * January 2016
 * December 2015
 * November 2015
 * October 2015
 * September 2015
 * August 2015
 * July 2015
 * June 2015
 * May 2015
 * April 2015
 * March 2015
 * February 2015
 * January 2015
 * December 2014
 * November 2014
 * October 2014
 * September 2014
 * August 2014
 * July 2014
 * June 2014
 * May 2014
 * April 2014
 * March 2014
 * February 2014
 * January 2014
 * December 2013
 * November 2013
 * October 2013
 * September 2013
 * August 2013
 * July 2013
 * June 2013
 * May 2013
 * April 2013
 * March 2013
 * February 2013
 * January 2013
 * December 2012
 * November 2012
 * October 2012
 * September 2012
 * August 2012
 * July 2012
 * June 2012
 * May 2012
 * April 2012
 * March 2012
 * February 2012
 * January 2012
 * December 2011
 * November 2011
 * October 2011
 * September 2011
 * August 2011
 * July 2011
 * June 2011
 * May 2011

 Categories

 * Amtrak
 * Cars
 * Consensus
 * Construction Costs
 * Development
 * Environmental Issues
 * FRA
 * Freight
 * Germany
 * Good Transit
 * Good/Interesting Studies
 * High-Speed Rail
 * Incompetence
 * Israel
 * Labor
 * New York
 * Pedestrian Observations
 * Personal/Admin
 * Politics and Society
 * Providence
 * Regional Rail
 * Shoddy Studies
 * Studies
 * Sweden
 * Transportation
 * Uncategorized
 * Urban Design
 * Urban Transit
 * Urbanism
 * Vancouver

 Meta

 * Register
 * Log in
 * Entries feed
 * Comments feed
 * WordPress.com

 Email Subscription

 Enter your email address to subscribe to this blog and receive notifications of new posts by email.

 Email Address:

 Sign me up!

 Join 1,877 other subscribers
 Blog at WordPress.com.
2023-01-30
	
	
	FDP and Vice Signaling	
	
	
	
	
Finance Minister Christian Lindner (FDP) just tweeted that more investment in roads is good – because if traffic flows more smoothly then there will be less greenhouse gas emissions. Reaction was not positive, and as of when I’m writing, 16 hours later, it is mildly ratioed. People understand that this is wrong. Lindner himself probably gets this too. Understanding what’s going on here requires talking about bullshit in the philosophical sense of Harry Frankfurt, and about something that I don’t have a better name for than vice signaling.
 Is it true? 
Absolutely not. It’s standard in transport studies that the construction of more highways in high-demand areas induces more traffic, as people take advantage of the greater convenience of driving. Drivers drive to new destinations that they forwent or chose to take public transport to, and new developments are built in areas opened by new highway development.
There may be exceptions to this in declining areas. The United States loves building new grade-separated interchanges in declining regions. This doesn’t generate new demand, because traffic is already uncongested, and the purpose of roadbuilding there is a political statement more than transport policy. But that’s not Germany. The roads under discussion here are in growth regions: there’s a plan to widen the beltway around Munich, A99, to 10 lanes, and the federal and Berlin FDP have both badgered Berlin to build a further stage of A100 parallel to the Ringbahn, which the city wants not to under the influence of the Green Party. Both motorway projects are likely to lead to adverse mode shift if built, and Lindner knows this.
There’s a developmental argument that induced demand is actually good. Matt Yglesias has made it before, saying that if road building induces more traffic then it means people get to take more trips and are better off. Many roadbuilders have made that very argument, and others were aware of it; Robert Moses, for example, was perfectly aware that his parkways and bridges were inducing more car traffic, and was fine with it, because he thought more driving was good. But that’s not what Lindner is saying: Lindner is saying that building new motorways and keeping them without a speed limit reduces greenhouse gas emissions, which is just bullshit.
 Bullshit 
The term “bullshit” has a precise meaning in analytic philosophy, due to Harry Frankfurt. It comprises a type of deception about the speaker’s mindset, rather than about the facts, unlike an ordinary lie. A politician who denies a scandal they are involved with is lying: their goal is to get you to believe that they are innocent of this scandal. A politician who, having been caught in said scandal, launches a series of schlock patriotic speeches is bullshitting: their goal is to get you to think they are fundamentally aligned with your values. From Frankfurt’s original essay, we have,
Telling a lie is an act with a sharp focus. It is designed to insert a particular falsehood at a specific point in a set or system of beliefs, in order to avoid the consequences of having that point occupied by the truth. This requires a degree of craftsmanship, in which the teller of the lie submits to objective constraints imposed by what he takes to be the truth. The liar is inescapably concerned with truth-values. In order to invent a lie at all, he must think he knows what is true. And in order to invent an effective lie, he must design his falsehood under the guidance of that truth. On the other hand, a person who undertakes to bullshit his way through has much more freedom. His focus is panoramic rather than particular. He does not limit himself to inserting a certain falsehood at a specific point, and thus he is not constrained by the truths surrounding that point or intersecting it. He is prepared to fake the context as well, so far as need requires. This freedom from the constraints to which the liar must submit does not necessarily mean, of course, that his task is easier than the task of the liar. But the mode of creativity upon which it relies is less analytical and less deliberative than that which is mobilized in lying. It is more expansive and independent, with mare spacious opportunities for improvisation, color, and imaginative play. This is less a matter of craft than of art. Hence the familiar notion of the “bullshit artist.”
The statement “widening roads reduces CO2 emissions” is this kind of bullshit. It is not quite a lie: it is false, but Lindner is not especially concerned with whether it is true or false. His goal is not to persuade people that building another section of A100 and widening A99 is good for climate; nobody who cares about climate change thinks that. Rather, his goal is to position himself as the sort of person who doesn’t listen to climate advocates and will just push for road widenings. The deception is part of the positioning: if he’d said that he understands the Greens’ argument against road investment but roads are important for economic development, he’d come off as too reasonable, which is not his intention.
Sounding deliberately unreasonable is the domain of populist politicians, and Frankfurt himself and many of his followers have noticed how political bullshit is on the rise as populism grows more normalized. Nigel Farage, for example, bullshitted that smoking isn’t bad for your health. And FDP is a populist party, despite its liberal origins and relatively moderate political positioning; it swung from deficit scold at the start of the current government to tax scold precisely as inflation rose last year, the opposite of what one should expect of a Washington Consensus-following economically orthodox party.
 Vice signaling 
There’s a pseudo-academic term going around the web, virtue signaling. The idea is that individuals and organizations engage in actions to signal that they’re better people than they really are; companies hire consultants on diversity, equity, and inclusion (DEI) without ever doing anything about their glass ceiling and harassment problems.
But it may be more fruitful to discuss its opposite – that is, vice signaling. This is when people take actions to portray themselves as terrible people, for any number of reasons:
Loyalty: criminal gangs are deliberately threatening and often require that prospective members commit murder (this is a requirement to become a made man in the Italian-American mafia), because this forces new members to have crossed both a moral and a legal event horizon from which they can’t come back; populist political movements don’t require crimes, but do require ridiculous beliefs
Novelty: this is what in the online language of the early 2010s was called the Slate Pitch – a take that aims to be novel by saying something really out there, often by writers who can’t separate themselves from the rest of the pack by any more productive means
Love of power: some people lie to you, with your full knowledge that they’re lying, just to flex that they can get away with it
Lindner loves this kind of vice signaling, I think out of novelty more than anything. FDP could be a party of YIMBYism, fiscal conservatism, and digital governance; younger members of the party who identify with neoliberalism wish that it were that party. The problem is that the difference between such a party and SPD is not large; Scholz ran on building more housing Germany-wide, and there’s a fair amount of consensus in favor of this in the party’s wings. SPD’s worst attributes so far are its officious leadership anchored in the Lower Saxony clique and consequently its sluggish governance and refusal to do more to support Ukraine – but FDP has the exact same problems, Lindner having told Ukraine when it asked for aid as the war started that there was no point since they’d fall in hours either way.
So to distinguish themselves from everyone else, FDP engages in vice signaling about climate and transport. They’re not trying to convince anyone that their policies are good for climate change. Rather, they’re doing the exact opposite: they’re trying to convince center-right voters that they’re an internal opposition within a coalition that is engaging in modal shift in federal funding priorities, and that they are explicitly against any climate action, because cars are good and only annoying hippies prefer trains.
	
	
	
	
	2023-01-28
	
	
	Quick Note: High Speed 2 and Euston	
	
	
	
	
There was reporting in the Sun, since officially denied, that Britain is planning to cut Euston from its high-speed rail project and run trains only as far a Old Oak Common, a future development site west of Central London. I assume given the source and lack of any other confirmation that the plans are to run to Euston as planned. But what if the story is not completely fake news, and there are plans to cut on construction at Euston? I can see a cut being positive value engineering, using space at the station more efficiently.
 What’s the issue with High Speed 2? 
High Speed 2 is an extremely expensive line. Among proper intercity high-speed rail lines (as opposed to suburban lines running at medium speed), it is the most expensive in our database per kilometer. The projected cost as of 2019-20 is about the same as that of all lines built to date in Germany and France combined; Germany has about 1,000 km of newly-built high-speed lines and France 2,500, whereas HS2 is planned to be 530 km.
The high costs are related to some massive scope creep, including tunnels in relatively flat terrain through the Chilterns, dug essentially because the area has rich NIMBYs and the British state decided not to fight them. Those are already in advanced enough construction that I don’t think descoping them and building the line at-grade with compulsory purchase of land is viable. However, some of the scope is new stations, which British defenders of the system insist are necessary. Birmingham is to get an entirely new station at Curzon Street, and London Euston is to get a substantial increase in size, with additional tracks and approaches. This is said to be necessary for capacity reasons.
 Are new stations necessary for capacity? 
No. Euston today has 16 platform tracks; it had 18 before HS2 construction started. The S-Bahn-quality Watford DC line can use two; the remaining slow services at the station amount to around 10-12 trains per hour, which S-Bahn-quality terminals like Saint-Lazare on the RER E and Catalunya on the Barcelona Rodalies network can comfortably turn on four tracks; those two comparisons turn 16 and 24 trains per hour on four tracks, respectively. The services out of Euston branch more than the RER and Rodalies, but this is mostly a mix of stopping patterns, largely on the same legacy line.
Then there’s HS2 itself. The line is expected to get very high ridership, justifiably: all cities along the line are larger than their comparison cases on the LGV Sud-Est, often substantially, and the projection is that very high capacity is required, on the order of 16 trains per hour. This stretches high-speed rail to its limit: the Shinkansen, which mixes local and express trains on double track, peaks around 14-15 trains per hour, and the complexly branched TGV around 12. HS2 expects to do better perhaps through better signals but also through having a simpler stopping pattern on its most congested section, between London and the bifurcation at Birmingham Interchange, on which trains are to run nonstop.
However, 16 trains per hour can still turn on about six platform tracks. This is not easy: the Tohoku Shinkansen turns 14 on four tracks, but this is a limit case, famous not just in rail media but also in business media as successful optimization of infrastructure. Nonetheless, given how constrained the site is, it’s useful to learn from the best and not the average. If it’s possible to descope the plans to add new tracks to Euston, this should be done; present plans for Euston cost billions.
 Is this happening? 
Maybe. Britain is aware of the situation at Tokyo Station, although it seems more interested in looking for reasons not to learn than in learning. Perhaps very high costs are leading to a reevaluation, in which Euston can be made smaller than in current plans and trains can turn more efficiently.
But again, the ultimate source on this said nothing of this sort, and is unreliable. So who knows?
	
	
	
	
	2023-01-26
	
	
	Free Public Transport, Fare Integration, and Capacity	
	
	
	
	
There’s an ongoing debate about free public transport that I’m going to get into later, but, for now, I want to zoom in on one aspect of the 9€ ticket, and how it impacted public transport capacity in Germany. A commenter on the Neoliberal Reddit group claimed that during the three months of nearly free public transport fares, there was a capacity crunch due to overuse. But in fact, the impact was not actually significant on urban rail, only on regional trains, in a way that underscores the importance of fare integration more than anything.
 What was the 9€ ticket? 
Last year, in the wake of the Russian invasion of Ukraine, fuel prices shot up everywhere. This created populist pressure to alleviate the price of fuel through temporary tax cuts, which further exacerbated last year’s high inflation. The center-right element within the German coalition, FDP, moved away from its traditional position as deficit scold and demanded a cut in the fuel tax; as a compromise, the Greens agreed to it on condition that during the three months of reduced fuel tax, June through August, public transport fares would be cut as well. Thus the 9€ monthly was born.
The 9€ ticket applied throughout Germany. The key feature wasn’t just the deep discount but also the fact that on one ticket, people could travel all over Germany; normally, my Berlin monthly doesn’t let me ride the local trains in Leipzig or Munich. This stimulated massive domestic tourism, since people could travel between cities on slow regional trains for free and then also travel around their destination city for free as well.
 What now? 
The 9€ ticket clearly raised public transport ridership in the three months it was in effect. This led to demands to make it permanent, running up against the problem that money is scarce and in Germany ticket fares generate a significant proportion of public transport revenue, 7.363 billion € out of 14.248 billion € in expenses (source, p. 36).
One partial move in that direction is a 29€ monthly valid only within Berlin, not in the suburbs (zone C of the S-Bahn) or outside the system; unlike the 9€ ticket, which was well-advertised all over national and local media and was available at every ticketing machine, the 29€ monthly is only available via annual subscription, which requires a permanent address in the city, and the regular machines only sell the usual 86€ monthly and don’t even let you know that a cheaper option exists. The subscription is also not available on a rolling basis – one must do it before the start of the month, which is not advertised, and Ant6n‘s family was caught unaware one month.
Negotiations for a nationwide 49€ ticket are underway, proceeding at the pace of a German train, or perhaps that of German arms deliveries to Ukraine. This was supposed to start at the beginning of 2023, then in April, and now it’s expected to debut in May. I’m assuming it will eventually happen – German trains get you there eventually, if hours late occasionally.
 What’s the impact on capacity? 
The U- and S-Bahn systems didn’t at all get overcrowded. They got a bit more crowded than usual, but nothing especially bad, since the sort of trips induced by zero marginal cost are off-peak. Rush hour commuters are not usually price-sensitive: whenever one’s alternative to the train is a car, the difference between a 9€ monthly and an 86€ one is a fraction of the difference between either ticket and the cost of owning and using a car, and at rush hour, cars are limited by congestion as well. Off-peak ridership did visibly grow, but not to levels that congest the system.
But then the hourly regional trains got completely overcrowded. If you wanted to ride the free trains from Berlin to Leipzig, you’d be standing for the last third of the trip. This is because the regional rail system (as opposed to S-Bahn) is designed as a low-capacity coverage-type system for connecting to small towns like Cottbus or Dessau.
The broader issue is that there is always a sharp ridership gradient between large cities and everywhere else, even per capita. In some places the gradient is sharper than elsewhere; the difference between New York and the rest of the United States is massive. But even in Germany, with a smaller gradient than one might be used to from France or the UK or Japan, public transport ridership is disproportionately dense urban or perhaps suburban, on trams and U- and S-Bahns.
The regional trains are another world. Really, European and Japanese trains can be thought of as three worlds: very high-use urban and suburban rail networks, high-use intercity rail connecting the main cities usually at high speed, and low-usage, highly-subsidized regional trains outside the major metropolitan regions. Germany has relatively good trains in the last category, if worse than in Switzerland, Austria, or the Netherlands: they run hourly with timed connections, so that people can connect between them to many destinations, they just usually don’t because cities the size of Dessau don’t generate a lot of ridership. The 9€ ticket gave people a free intercity trip if they chained trips on these regional trains, at the cost of getting to Leipzig in a little less than three hours rather than 1:15 on the ICE; the regional trains were not expanded to meet this surge traffic, which is usually handled on longer intercity trainsets, creating standing-room only conditions on trains where this should not happen off-peak or perhaps ever.
 The issue of fare integration 
The overcrowding seen on the regional trains last summer is really an issue of fare integration, which I hope is resolved as the 49€ gives people free trips on such trains permanently. A cornerstone of good public transport planning is that the fare between two points should be the same no matter what vehicle one uses, with exceptions only for first-class cars if available. Ein Ticket für alles, exclaims the system in Zurich, to great success. Anything else slices the market into lower-frequency segments, providing worse service than under total fare integration. Germany understands this – the Verkehrsverbund was invented in Hamburg in 1965, and subsequently this idea was adopted elsewhere until the country has been divided into metropolitan zones with internal fare integration.
The regional trains that cross Verkehrsverbund zones have their own fares, and normally that’s okay. Intercity trains were never part of this system, and that’s okay too – they’re not about one’s usual trip, and so an intercity ticket doesn’t include free transfers to local public transport unless one pays extra for that amenity. The fares between intercity trains and chains of regional trains were not supposed to be integrated, and normally that’s fine too, because any fare savings from chaining trips on slower trains are swamped both by the headache of buying so many tickets and by the difference in trip time and reliability.
The 9€ ticket broke that system, and the 49€ ticket will have the same effect: for three months, trips on slower trains were free, leading to overcrowding on a low-capacity network that normally isn’t that important to the country’s overall public transport system.
Worse, the operating costs of slow trains are higher than those of fast trains: they are smaller and so have a higher ratio of crew to passengers than ICEs, and their slowness means that crew and maintenance costs per kilometer are higher than those of fast trains. Even energy costs are higher on slow trains, because high-speed lines run at 300 km/h over long stretches, whereas regional lines make many stops (which had very little usage compared with the train’s volume of passengers last summer) and have slow zones rather than cruising at 130 or 160 km/h over long stretches. So the system gave people a price incentive to use the higher-cost trains and not the lower-cost ones.
This is the most important thing to resolve about any future fare reductions. Some mechanism is needed to ensure that the most advantageous way to travel between two cities is the one that DB can provide the most efficiently, which is IC/ICE and not RegionalBahn.
	
	
	
	
	2023-01-25
	
	
	Our Construction Costs Report	
	
	
	
	
This is awkward.
I was hoping to already release the Transit Costs Project report, but we ran into difficulties after we released and then immediately unreleased; we sent the report to the MTA for feedback, we got feedback, and then we went on a binge of edits. This led to a long cycle of almosts; I thought it would be done at the end of December, then 10 days ago, then tomorrow. The revisions were not huge – there are people who read the original version and the gap between what they’ve seen and what we’re about to release is real but is also far from the difference between complete knowledge (which we don’t have and most likely never will) and complete ignorance.
We still have minor edits to do, but they’re small, on the order of less than a full day’s work for all three of us, and we can promise that the full report – including the New York report and the synthesis of all reports combined – will be out on February 6th.
Separately, I’m going to be in New York for about a month after, mostly for non-work reasons. The timing is that there’s an election in Berlin on the 12th, a redo of the city election from 2021 (oddly, not electing for a full term but for the remainder of the term that began in 2021), in which I hope the Greens do better if only so that they’re slightly ahead of SPD and not slightly behind and can replace the scandalized Franziska Giffey as mayor. If I can vote by mail, I’ll be in New York starting a little before the election; if I can’t, I’ll be starting a little later. This will include at least one more in-person event to present our findings and our recommendations for the region and the MTA, at a time to be determined but likely in the second half of February.
There’s a lot of stuff we didn’t get into the report, because of lack of space; we could have info-dumped and made the synthesis novel-length, but some things had to go (and I will defend every decision to exclude things, and if they turn out to have been necessary, then blame me for being wrong about it). So even if you read the reports you should definitely come to events and ask me specific questions about things that are related to construction costs, such as other projects in the same cities we studied, or issues of the interplay between costs and benefits, or how this applies to urban rail construction going forward.
	
	
	
	
	2023-01-24
	
	
	Push and Pull Factors and Measuring Modal Shift	
	
	
	
	
There’s a longstanding debate among activists and academics about what the best way of effecting modal shift from cars to public transport is. Pull factors concern making public transport better through building more rail lines, running them more frequently, improving service convenience, or reducing fares. Push factors concern making driving harder through speed limits, fuel taxes, congestion pricing, and reallocation of street space from cars to public and non-motorized transport. There’s a tendency on the New Left to favor push factors (but the East Asian developmental states are best characterized as push-before-pull and not pure pull).
This has been refined by researchers at the climate research institute, the Ariadne Project, who published a paper in late 2021 rating various push and pull policies on effectiveness for reducing transport emissions. They conclude that push factors dominate, and pull factors are small, with construction of new public transit almost insignificant, only worth a reduction of around 300,000 tons of CO2 a year Germany-wide, 0.039% of national emissions as of 2021; instituting a 120 km/h speed limit on the Autobahn is said to have about 10 times that effect, while the biggest effects yet would come from carbon taxes. The study laments that pull factors are so much more popular than push factors, which they admit suppress society-wide consumption.
The research suffers from the same problem as other work in this direction, in that it is bad at estimating the impact of public transport on mode shift. It briefly argues that construction of public transport increases overall consumption and therefore doesn’t do much to reduce emissions. This way, it’s like 2020’s carbon critique of U-Bahn expansion, which I criticized two months ago; the carbon critique argues that each kilometer of U-Bahn built only reduces CO2 emissions by 714 tons a year through mode shift, under the assumption that only 20% of public transport riders are diverted from cars.
This doesn’t pass a sanity check. 300,000 divided by 714 is 420 km, which is about comparable to the total route length of the four grade-separated U-Bahn systems in Germany plus the Wuppertal Schwebebahn; I think the two figures, 300,000 and 714/km, come from different sources, and judging by the other elements in the study, I suspect 300,000 assumes less construction than a full doubling of Germany’s rapid transit network length. Nonetheless, even under a more generous assumption, this is far too low compared with macro trends in public transport usage.
The best way to use macro trends as a sanity check is to look at some cases with much more and much less public transport than the present. Do they look like it’s a total difference of 0.039%? No, and that’s even taking into account that transit cities tend to be wealthier, stimulating more consumption and more production. As I pointed out in my post two months ago, while Germany averages 9.15 t-CO2/capita, Berlin only does 5.38, and while Germany averages 580 cars per 1,000 people, Berlin only does 327. The difference is largely about Berlin’s pull factors. Push factors in the city are not extensive, and what exists is implemented only in areas that already have very low car use.
Even lower household emissions in Berlin must be viewed as downstream of the density that is enabled by the presence of a large urban rail network. Cars are a low-capacity mode of transport, so an auto-oriented region, like American metro regions, has to spread out its homes and destinations to limit congestion, and this increases household emissions (single-family houses emit more than apartment buildings) and also encourages people to travel longer distances for their commute and routine non-commute trips.
This is not easy to measure. Public transport projects have gotten fairly good in the last generation at estimating ridership, but estimating the responsibility of one particular project to modal shift is hard. It interacts with the entire city region. For example, building one rail line can be measured to shift modes in the neighborhoods it serves, but it also encourages destinations to locate in city center since people from the neighborhoods the line serves can now access it, and the increase in office, retail, and community development then leads to a small modal shift citywide. Worse, trying to tease out the effect of the rail line on modal shift sufficiently carefully may lead researchers to count this citywide effect negatively, since one econometric technique is to compare the neighborhoods near the line with neighborhoods in the same city not on the line.
In practice, the construction of rail lines tends to co-occur with other policies that improve public transport, which may be pull or push factors. This means that it’s very easy to misattribute the effect of urban rail expansion to those other factors. I am convinced that this is what is happening here; the proper comparison must be at the level of an entire region, looking at the emissions of different regions with different levels of public transport usage.
The upshot is that if it is hard to measure the effect of public transport construction on modal shift and emissions, then the uncertain factors should not be set to zero. Rather, they should be set to sanity-check levels. For example, one can compare New York with the rest of the United States, since it’s a starker difference between a transit city and an auto-oriented country than anywhere in Europe, and correct for non-transport effects like climate and electricity mix, both of which are easy to measure.
Within Germany, Berlin has 42% lower emissions than the rest of the country per capita. Berlin achieves this with an urban rail network that, in 2019, got 1,289 million rail trips, nearly all within the city of 3.7 million, a minority in the suburban region of perhaps 1.3 million. This is around 250 trips/person regionwide, and 320/person citywide assigning around 20% of S-Bahn ridership to suburbs like Potsdam and Oranienburg. What’s more, Germany doesn’t start from zero; this is not the United States, with multiple large cities with around 10 annual rail trips per capita. Netting out buses from VDV’s data (p. 25) gets around 6.3 billion rail trips in Germany in 2019 including trams, or 75 per capita.
The difference between 320 and 75 is around 250 – I know it’s actually 245 but at this point I’m deliberately reducing precision because those are sanity-check estimates and I don’t want people thinking they’re correct to three significant figures (try 1.5). If we attribute the entire Berlin-Germany difference of about 3.8 t-CO2/capita to public transport and downstream changes to the urban layout, then we get 0.015 t saved per annual trip generated. To get from there to 300,000 tons saved, we just need 20 million annual rail riders, or around 65,000 daily ones, which is easy to generate on a single line; the approximately 2 km extension of U8 to Märkisches Viertel that Berlin keeps postponing is estimated to generate 25,000-30,000.
Now, to sanity-check the sanity check, the estimate here is that every trip on urban rail saves 15 kg-CO2. This is an aggressive figure; new cars nowadays average 100 g/km and averaged 180 g/km in 2001 (source, PDF-p. 15), and the average displaced car trip is not 150 km or even 80 km – Americans average around 45 km/day, or somewhat more when only adults are considered. Rather, the issue is a combination of factors:
Because the limiting factor to car transport is capacity, in practice what happens in an auto-oriented region is that it fills from the inside outward, and any modal shift ends up displacing the outermost and longest car trips. I proposed a model for that in a blog post from four years ago.
Public transport displaces car trips on a more than one-to-one basis (and certainly more than 20% as in the carbon critique of the U-Bahn). This is because public transport users also walk and bike, and transit cities have high modal splits for active transport by the standards of auto-oriented cities, if not by the standards of Dutch cities. Berlin’s all-trip modal split in 2018 was 26% car, 27% public transport, 18% bike, 30% walking – and the high active transport modal split exists not because of road diets, which are few and far between, but because of the presence of a large core fed by the U- and S-Bahn.
Public transport reduces household energy usage by encouraging people to live in apartment buildings with shared walls rather than in single-family houses, which have much greater heating requirements; this is also the mechanism through which transit cities have relatively high usage of active transport even without trying very hard.
I don’t think these factors fully explain away the gap between 45 km/day and 150 km per trip (so around 300/day), but they explain a large enough fraction of it that the installation of a system like what Berlin has – or, better, what Tokyo has – should be a climate priority. If your model says it doesn’t, it needs a lot more work than to just talk about the consumption effects of more public transport (if you’re bothered by how Berlin is poor for its size, compare New York with the rest of the United States).
In fact, if estimating modal shift is hard, then it’s best to approximate it by ridership. It’s imperfect because there is the effect of walking and biking; some lines really do just compete with walking, like city-center streetcars, but usually, to first order, it’s a good enough estimate. If it’s hard to estimate the benefits then they should not be set to zero, but rather set proportionally to something easier to measure, in this case ridership. Investment should follow ridership-maximizing strategies, and only deviate from them in corner cases.
	
	
	
	
	2023-01-22
	
	
	Paris-Berlin Trains, But no Infrastructure	
	
	
	
	
Yesterday, Bloomberg reported that Macron and Scholz announced new train service between Paris and Berlin to debut next year, as intercity rail demand in Europe is steadily rising and people want to travel not just within countries but also between them. Currently, there is no direct rail service, and passengers who wish to travel on this city pair have to change trains in Frankfurt or Cologne. There’s just one problem: the train will not have any supportive infrastructure and therefore take the same eight hours that trains take today with a transfer.
This is especially frustrating, since Germany is already investing in improving its intercity rail. Unfortunately, the investments are halting and partial – right now the longest city pair connected entirely by high-speed rail is Cologne-Frankfurt, a distance of 180 km, and ongoing plans are going to close some low-speed gaps elsewhere in the system but still not create any long-range continuous high-speed rail corridor connecting major cities. With ongoing plans, Cologne-Stuttgart is going to be entirely fast, but not that fast – Frankfurt-Mannheim is supposed to be sped up to 29 minutes over about 75 km.
Berlin-Paris is a good axis for such investment. This includes the following sections:
Berlin-Halle is currently medium-speed, trains taking 1:08-1:16 to do 162 km, but the flat, low-density terrain is easy for high-speed rail, which could speed this up to 40-45 minutes at fairly low cost since no tunnels and little bridging would be required.
Halle-Erfurt is already fast, thanks to investments in the Berlin-Munich axis.
Erfurt-Frankfurt is currently slow, but there are plans to build high-speed rail from Erfurt to Fulda and thence Hanau. The trip times leave a lot to be desired, but newer 300 km/h trains like the Velaro Novo, and perhaps a commitment to push the line not just to Hanau but closer to Frankfurt itself, could do this section in an hour.
Frankfurt-Saarbrücken is very slow. Saarbrücken is at the western margin of Germany and is not significant enough by itself to merit any high-speed rail investment. Between it and Frankfurt, the terrain is rolling and some tunneling is needed, and the only significant intermediate stops are Mainz (close enough to Frankfurt it’s a mere stop of opportunity) and Kaiserslautern. Nonetheless, fast trains could get from Frankfurt to the border in 45 minutes, whereas today they take two hours.
Unfortunately, they’re not talking about any pan-European infrastructure here. Building things is too difficult, so instead the plan is to run night trains – this despite the fact that Frankfurt-Saarbrücken with a connection to the LGV Est would make a great joint project.
	
	
	
	
	2023-01-19
	
	
	Bad Public Transit in the Third World	
	
	
	
	
There’s sometimes a stereotype that in poor countries with low car ownership, alternatives to the car are flourishing. I saw a post on Mastodon making this premise, and pointed out already in comments that this is not really true. This is a more detailed version of what I said in 500 characters. In short, in most of the third world, non-car transportation is bad, and nearly all ridership (on jitneys and buses) is out of poverty, as is most walking. While car ownership is low, the elites who do own cars dominate local affairs, and therefore cities are car-dominated and not at all walkable, even as 90%+ of the population does not own a car.
What’s more, the developing countries that do manage to build good public transportation don’t stay developing for long. The same development model of Japan, the East Asian Tigers, and now China has built both rail-oriented cities and high economic growth, to the point that Japan and the Tigers are fully developed, and China is a solidly middle-income economy. The sort of places that stay poor, or get stuck in a middle-income trap, also tend to have stagnant urban rail networks, and so grow more auto-oriented over time. 
 The situation in Southeast Asia 
With the exception of Singapore, nowhere in Southeast Asia is public transit good. What’s more, construction costs have been high for elevated lines and very high for underground ones, slowing down the construction of metro systems.
In Kuala Lumpur and Bangkok, motorization is high and public transit usage is weak. Paul Barter’s thesis details how both cities got this way, in comparison with the more transit-oriented model used in Tokyo, Seoul, Hong Kong, and Singapore. The thesis also predicts that the poorer megacities of Southeast Asia – Jakarta and Manila – will follow the auto-oriented path as they develop, which has indeed happened in the 13 years since it was written.
The situation in those cities is, to be fair, murky. Manila has a large urban rail under construction right now, with average to above average costs for elevated lines and high ones for subways. But the system it has today consists of four lines, two branded light rail, one branded MRT, and one commuter line. In 2019, the six-month ridership on the system was 162 million. A total of 324 million in a metro area the size of Manila is extraordinarily low: the administrative Metro Manila region has 13.5 million people, and the urban or metropolitan area according to both Citypopulation.de and Demographia is 24-26 million. On the strictest definition of Metro Manila, this is 24 trips per person per year; on the wider ones, it is about 13, similar to San Diego or Portland and only somewhat better than Atlanta.
Jakarta is in the same situation of flux. It recently opened a half-underground MRT line at fairly high cost, and is modernizing its commuter rail network along Japanese lines, using second-hand Japanese equipment. Commuter rail ridership was 1.2 million a day last year, or around 360 million a year, already higher than before corona; the MRT had 20 million riders last year, and an airport link had 1.5 million in 2018. This isn’t everything – there’s also a short light metro called LRT for which I can’t find numbers – but it wouldn’t be more than second-order. This is 400 million annual rail trips, in a region of 32 million people.
The future of these cities is larger versions of Bangkok. Thailand is sufficiently middle-income that we can see directly how its transport system evolves as it leaves poverty, and the results are not good. Bus ridership is high, but it’s rapidly falling as anyone who can afford a car gets one; a JICA report about MRT development puts the region’s modal split at 5% MRT, 36% bus, and the rest private (PDF-p. 69) – and the income of bus riders is significantly lower than that of drivers (PDF-p. 229), whereas MRT riders are closer to drivers.
Even wealthier than Bangkok, with the same auto-oriented system, is Kuala Lumpur. There, the modal split is about 8% bus, 7% train, and the rest private. This is worse than San Francisco and the major cities of Canada and Australia, let alone New York or any large European city. The national modal split in England, France, Germany, and Spain is about 16% – the first three countries’ figures predate corona, but in Spain they’re from 2021, with suppressed public transport ridership. Note that rail ridership per capita is healthier in Kuala Lumpur than in Jakarta or Manila – all rail lines combined are 760,000 riders per day, say 228 million per year, in a region of maybe 7 million people. This is better than a no-transit American city like San Diego, but worse than a bad-transit one like Chicago or Washington, where the modal split is about the same but there is no longer the kind of poverty that is common in Malaysia, let alone in Indonesia, and therefore if people ride the trains it’s because they get them to their city center jobs and not because they’re poor.
Even in Singapore, the best example out there of a transit-oriented rich city, it took until very recently for MRT coverage to be good enough that people willingly depend on it; it only reached NUS after I graduated. In the 1990s, the epitome of middle-class Singaporean materialism was described as owning the Five Cs, of which one was a car; traffic suppression, a Paul Barter describes, has centered fees on cars, much more car purchase than car use (despite the world-famous congestion pricing system), and thus to those wealthy enough to afford cars, they’re convenient in ways they are not in Paris, Berlin, or Stockholm.
 The situation in Africa 
African countries between the Sahara and the Kalahari are all very poor, with low car ownership. However, they are thoroughly car-dominated.
From the outside, it’s fascinating to see how the better-off countries in that region, like Nigeria, are already imitating Southeast Asia. Malaysia overregulated its jitneys out of existence because they were messy and this bothered elites, and because it wanted to create an internal market for its state-owned automakers. Nigeria is doing the same, on the former grounds; to the extent it hasn’t happened despite years of trying, it’s because the state is too weak to do more than harass the drivers and users of the system.
It’s notable that the Lagos discourse about the evils of the danfo – they are noisy, they are polluting, they drive like maniacs – there is little attention to how cars create all the same problems, except at larger scale per passenger served. The local notables drive (or are driven); the people who they scorn as unwashed, overly fecund, criminal masses ride the danfo. Thanks to aggressive domination by cars and inattention to the needs of the non-driving majority, Lagos’s car ownership is high for how poor it is – one source from 2017 says 5 million cars in the state, another from 2021 says 6.5 million vehicles between the state and Kano State. The denominator population in the latter source is 27 million officially, but unofficially likely more; 200 vehicles per 1,000 people is plausible for Lagos, which to be clear is not much less than New York or Paris, on an order of magnitude lower GDP per capita. Tokyo took until about 1970 to reach 100 vehicles per 1,000 people, at which point Japan had almost fully converged with American GDP per capita.
This is not specific to Lagos. A cousin who spent some time in Kampala told me of the hierarchy on the roads: pedestrians fear motorcycles, motorcycles fear cars, cars fear trucks. There is no pedestrian infrastructure to speak of; a rapid transit system is still a dream, to the point that a crayon proposal that spread on Twitter made local media. That the vast majority of Ugandans don’t own cars doesn’t matter; Kampala remains dominated by the few who do.
 Transit and development 
I don’t think it’s a coincidence that the sort of developing countries that build successful urban rail systems don’t stay poor for long. Part of it is that public transportation is good for economic development, but that’s not most of it – the United States manages to be rich without it except in a handful of cities. Rather, I suspect the reason has to do with state capacity.
More specifically, the reason cities with 100-200 cars per 1,000 people are thoroughly dominated by cars is that those 10-20% drivers (or people who are driven) are the elites. Their elite status can come from any source – passive business income, landlordism, active business income, skilled professional work – but usually it tilts toward the traditional, i.e. passive. These groups tend to be incredibly anti-developmental: they own small businesses, sometimes actively and sometimes passively, and resent being made redundant through economies of scale. India has problems with economic dwarfism and informality, and this is typical of poor countries; if anything, India is better than most at developing a handful of big businesses in high-value added industries.
The upshot is that the sort of people who drive, and especially the sort of drivers who are powerful enough to effect local changes to get incremental upgrades to roads at the expense of non-drivers, are usually anti-developmental classes. The East Asian developmental states (and Singapore and Hong Kong, which share many characteristics with them) clamped down on such classes hard, on either nationalist or socialist grounds; Japan, both Koreas, and both Chinas engaged in land reform, with characteristic violence in the two socialist states and without it but still with forcible purchase in the three capitalist states. The same sort of state that can eliminate landlordism can also, as a matter of capital formation, clamp down on consumption and encourage personal savings, producing atypically low levels of motorization well into middle-income status. Singapore, whose elite consumption centers vacations out of the country, has managed to do so even as a high-income country – and even more normal Tokyo and Seoul have much higher rail usage and lower car usage than their closest Western analog, New York.
India is in many ways anti-developmental, but it does manage to grow. Its anti-developmentalism is anti-urban and NIMBY, but it is capable of building infrastructure. Its metro program has problems with high construction costs (but Southeast Asia’s are generally worse) and lack of integration with other modes such as commuter rail, which the middle class denigrates as only befitting poor people; but the Delhi Metro had 5.5 million daily riders just before corona, slightly behind New York in a slightly larger metro area, perhaps a better comparison than Jakarta and Manila’s San Diego.
It’s the slower-growing developing countries that are not managing to even build the systems India has, let alone East Asia. They don’t have high car use, but only because they are poor, and in practice, they are thoroughly car-dominated, and everyone who doesn’t have a car wants one. A rich country really is not one where even the poor have cars but where even the rich use public transportation – and those countries aren’t rich and don’t grow at rates that will make them rich.
	
	
	
	
	2023-01-12
	
	
	High Costs are not About Precarity	
	
	
	
	
I’ve seen people who I think highly of argue that high construction costs in the United States are an artifact of precarity. The argument goes that the political support for public transportation there is so flimsy that agencies are forced to buy political support by spending more money than they need. This may include giving in to NIMBY pressure to use costlier but less impactful (or apparently less impactful) techniques, to spread money around with other government agencies and avoid fighting back, to build extravagant and fancier-looking but less standardized stations, and so on. The solution, per this theory, is to politically support public transportation construction more so that transit agencies will have more backing.
This argument also happens to be completely false, and the solution suggested is counterproductive. In fact, the worst cost blowouts are for the politically most certain projects; Second Avenue Subway enjoyed unanimous support in New York politics.
 Cost-effectiveness under precarity 
Three projects relevant to our work at the Transit Costs Project have been done exceptionally cost-effectively in an environment of political uncertainty: the T-bana, the LGV Sud-Est, and Bahn 2000.
 T-bana 
The original construction of the T-bana was done at exceptionally low cost. We go over this in the Sweden report to some extent, but, in short, between the 1950s and 70s, the total cost of the system’s construction was 5 billion kronor in 1975 prices, which built around 100 km, of which 57% are underground. In PPP 2022 dollars, this is $3.6 billion, or $35 million/km, not entirely but mostly underground. This was low for the time: for example, in London, the Victoria line was $122 million/km and the Jubilee line was $172 million/km (source, p. 78), and Italian costs in the 1960s and 70s were similar, averaging $129 million/km before 1970.
The era of Social Democrat dominance in Swedish politics on hindsight looks like one of consensus in favor of big public projects. But the T-bana itself was controversial. When the decision was made to build it in the 1940s, Stockholm County had about 1 million people; at the time, metros were present in much larger cities, like New York, London, Paris, Berlin, and Tokyo, and it was uncertain that a city the size of Stockholm would need such a system. Its closest analog, Copenhagen, did not build such a system until the 1990s, when it was a metro region of 2 million. It was uncertain that Stockholm should need rapid transit, and there were arguments for and against it in the city. Nor was there any transit-first policy in postwar Sweden: urban planning was the same modernist combination of urban renewal, automobile scale, and tower-in-a-park housing, and outside Stockholm County, the Million Program projects were thoroughly car-oriented.
Construction costs in Sweden are a lot higher now than they were in the 1950s, 60s, and 70s. Nya Tunnelbanan is $230 million/km, compared with a post-1990s Italian average of $220 million; British costs have exploded in tandem, so that now the Underground extensions clock at $600 million/km. Our best explanation is that the UK adopted what we call the globalized system of procurement, privatizing planning functions to consultants and privatizing risk to contractors, which creates more conflict; the UK also has an unusually high soft cost factor. From American data (and not just New York) and some British data, I believe that the roughly 2.5 cost premium of the UK over Italy is entirely reducible to such soft costs, procurement conflict, risk compensation, and excessive contingency. And yet, Sweden itself, with some elements of the same globalized system, maintains a roughly Italian cost level, albeit trending the wrong way.
And today, too, the politics of rail expansion in Sweden are uncertain. There was controversy over both Citybanan and Nya Tunnelbanan, neither of which passed a cost-benefit analysis (for reasons that I believe impugn the cost-benefit analysis more than those projects); it was uncertain that either would be funded. Controversy remained over plans to build high-speed rail connecting Stockholm with Gothenburg and Malmö, and the newly-elected right-wing government just canceled them in order to prioritize investment in roads. Swedish rail projects today remain precarious, and have to justify themselves on cost and efficiency grounds.
 LGV Sud-Est 
Like nearly all other rich countries, France was hit hard by the 1973 oil crisis; economic growth there and in the US, Japan, and most of the rest of Western Europe would never be as high as it was between the end of WW2 and the 1970s (“Trente Glorieuses“). On hindsight, France’s response to the crisis models can-go governance, with an energy saving ad declaring “in France we don’t have oil, but we have ideas.” The French state built nuclear power plants with gusto, peaking around 90% of national electricity use – and even today’s reduced share, around 70%, is by a large margin the highest in the world. At the same time, it built a high-speed rail network, connecting Paris with most other provincial cities at some of the highest average speeds outside China between major cities, reaching about 230 km/h between Paris and Marseille and 245 km/h between Paris and Bordeaux; usage per capita is one of the highest in Europe and, measured in passenger-km, not too bad by East Asian standards.
But in fact, the first LGV, the LGV Sud-Est, was deeply controversial. At the time, the only high-speed rail network in operation was the Shinkansen, and while France learned more from Japan than any other European country (for example, the RER was influenced by Tokyo rail operations), the circumstances for intercity were completely different. SNCF had benefited from having done many of its own experiments with high-speed technology, but the business case was murky. SNCF had to innovate in running an open system, with extensive through-running to cities off the line, which Japan would only introduce in the 1990s with the Mini-Shinkansen.
Within the French state, the project was controversial. Anthony Perl’s New Departures details how there were people within the government who wanted to cancel it entirely as it was unaffordable. At the end, the French state didn’t finance the line, and required SNCF to find private loans on the international market, though it did guarantee those loans. It also delayed the line’s opening: instead of opening the entire line from Paris to Lyon in one go, it opened two-thirds of it on the Lyon side in 1981 and the last third into Paris in 1983, requiring trains to run on the classical line at low speed between Paris and Saint-Florentin for two years; in that era, phased opening was uncommon, and lines generally opened to the end at once, such as between Tokyo and Shin-Osaka.
Construction was extraordinarily inexpensive. In PPP 2022 dollars, it cost $8.4 million/km. This is, by a margin, the lowest-cost high-speed rail line ever built that I know about. The Tokaido Shinkansen cost  380  billion yen, or in PPP 2022 dollars $40 million/km, representing a factor of two cost overrun that forced JNR’s head to resign. Spain has unusually low construction costs, and even there, Madrid-Seville was $15.7 million/km. SNCF innovated in every way possible to save money. Realizing that high-speed trains could climb steeper grades, it built the LGV Sud-Est with a ruling grade of 3.5%, which has since become a norm in and around Europe, compared with the Shinkansen’s 1.5-2%; the line has no tunnels, unlike the classical Paris-Lyon line. It built the line on the ground rather than on viaducts, and balanced cut and fill locally so that material cut to grade the line could be used for nearby fill. Thanks to the line’s low costs and high ridership, the financial return on investment for SNCF has been 15%, and social return on investment has been 30% (source, pp. 11-12).
This cost-effectiveness would never recur. The line’s success ensured that LGV construction would enjoy total political backing. The core features of LGV construction are still there – earthworks rather than viaducts, 3.5% grades, limited tunneling, overcompensation of landowners by about 30% with land swap deals to defuse the possibility of farmer riots. But the next few lines cost about $20 million/km or slightly less, and this cost has since crept up to about $30 million/km or even more. This remains low by international standards (but not by Spanish ones), but the trend is negative.
SNCF is coasting on its success from a generation ago, secure that funding for LGVs and state support in political contention is forthcoming, and the routing decisions have grown worse. In response to NIMBYism in Provence, the French state assented to a tunnel-heavy route, including a conversion of Marseille from an at-grade terminal to an underground through-station, akin to Stuttgart 21, which has not been done before in France, and the resulting high costs have led to delays on the project. Operations have grown ever more airline-style, experimenting with low-cost airline imitation to the point of reducing fare receipts without any increase in ridership. One of the French consultants we’ve spoken with said that their company’s third-party design costs are 7-8% of the hard costs, which figure is similar to what we’ve seen in Italy and to the in-house rate in Spain – but the same consultant told us that there is so much bloat at SNCF that when it designs its own projects, the costs are not 7% but 25%, a figure in line with American rates.
 Bahn 2000 
Switzerland has Europe’s strongest passenger rail network by all measures: highest traffic measured by passenger-km per capita, highest modal split for passenger-km, highest traffic density. Its success is well-known in surrounding countries, which are gradually either imitate its methods or, in the case of Germany, pretending to do so. It has achieved its success through continuous improvement over the generations, but the most notable element of this system was implemented in the 1990s as part of the Bahn 2000 project.
The current system is based on a national-scale clockface system (“Takt”) with trains repeating hourly, with the strongest links, like the Zurich-Bern-Basel triangle, running every half hour. Connections are timed in those three cities and several others, called knots, so that trains enter each station a few minutes before the connection time (usually the hour) and depart a few minutes after, permitting passengers to get between most pairs of Swiss cities with short transfers. Reliability is high, thanks to targeted investments designed to ensure that trains could make those connections in practice and not just in theory. Further planning centers adding more knots and expanding this system to the periphery of Switzerland.
Switzerland is famous for its consensus governance system – its plural executive is drawn from the four largest parties in proportion to their votes, with no coalition vs. opposition politics. But the process that led to the decision to adopt Bahn 2000 was not at all one of unanimity. There had been plans to build high-speed rail, as there were nearly everywhere else in Western Europe. But they were criticized for their high costs, and there was extensive center-right pressure to cut the budget. Bahn 2000 was thus conceived in an environment of austerity. Many of its features were explicitly about saving money:
The knot system is connected with running trains as fast as necessary, not as fast as possible. Investments in speed are pursued only insofar as they permit trains to make their connections; higher speeds are considered gratuitous.
Bilevel trains are an alternative to lengthening the platforms.
Timed overtakes and meets are an alternative to more extensive multi-tracking of lines.
Investment in better timetabling and systems (the electronics side of the electronics-before-concrete slogan) is cheaper than adding tunnels and viaducts.
Swiss megaprojects have to go to referendum, and sometimes the referendums return a no; this happened with the Zurich U-Bahn twice, leading to the construction of the S-Bahn instead. All Swiss planners know in a country this small and this fiscally conservative, any extravagance will lead to rejection. The result is that they’ve instead optimized construction at all levels, and even their unit costs of tunneling are low; thanks to such optimization, Switzerland has been able to build a fairly extensive medium-speed rail system, with more tunneling per capita than Germany (let alone France), and with two S-Bahn trunk tunnels in Zurich, where no German city today has more than one.
 The American situation 
The worst offenders in the United States are not at all politically precarious. There is practically unanimous consensus in New York about the necessity of Second Avenue Subway. At no point was the project under any threat. There is an ideological right in the city, rooted less in party politics and more in the New York Post and the Manhattan Institute, with a law-and-order agenda and hostility to unions and to large government programs, but at no point did they call for cancellation; the Manhattan Institute’s Nicole Gelinas has proposed pension cuts for workers and rule changes reducing certain benefits, but not canceling Second Avenue Subway.
At intercity scale, the same is true of Northeast Corridor investment. The libertarian and conservative pundits who say passenger rail is a waste of money tend to except the Northeast Corridor, or at least its southern half. When the Republicans won the 2010 midterm election, the new chair of the House of Representatives Transportation Committee, John Mica (R-FL), proposed a bill to seek private concessionaires to run intercity rail on the corridor. He did not propose canceling train service, even though in the wake of the same election, multiple conservative governors canceled intercity rail investments in their state, both high-speed (Florida) and low-speed (Wisconsin, Ohio).
In fact, both programs – New York subway expansion and the Northeast Corridor – are characterized by continuity across partisan shifts, as in more established consensus governance systems. The Northeast Corridor is especially notable for how little role ideological or partisan politics has played so far. New York has micromanagement by politicians – Andrew Cuomo had his pet projects in Penn Station Access and the backward-facing LaGuardia air train, and now Kathy Hochul has hers in the Interborough Express – but Second Avenue Subway was internal, and besides, political micromanagement is a different problem from political precarity.
And neither of these programs has engaged in any cost control. To the contrary, both are run as if money is infinite. The MTA would surrender to NIMBYs (“good neighbor policy”) and to city agencies looking to extract money from it. It built oversize stations. It spent money protecting buildings from excessive settlement that have been subsequently demolished for redevelopment at higher density.
The various agencies involved in the Northeast Corridor, likewise, are profligate, and not for lack of political support. Connecticut is full of NIMBYs; one of the consultants working on the plan a few years ago told me there was informal pressure not to ruffle feathers and not to touch anything in the wealthiest suburbs in the state, those of Fairfield County. In fact, high-speed rail construction would require significant house demolitions in the state’s second wealthiest town, Darien – but Darien is so infamously exclusive (“Darien rhymes with Aryan,” say other suburbanites in the area) that the rest of the region feels little solidarity with it.
NIMBYs aside, there has not been any effort at coordinating the different agencies in the Northeast along anything resembling the Swiss Takt system. This is not about precarity, because this is not a precarious project; this is about total ignorance and incuriosity about best practices, which emanate from a place that doesn’t natively speak English and doesn’t trade in American political references.
 The Green Line Extension 
Boston’s GLX is a fascinating example of cost blowouts without precarity. The history of the project is that its first iteration was pushed by Governor Deval Patrick (D), with the support of groups that sued the state for its delays in planning the project in the 1990s, as a court-mandated mitigation for the extra car traffic induced by the construction of the Big Dig. Patrick instituted a good neighbor policy, in which everything a community group wanted, it would get. Thus, stations were to become neighborhood signatures, and the project was laden with unrelated investments, called betterments, like a $100 million 3 km bike lane called the Somerville Community Path.
At no point in the eight years Patrick was in power was there a political threat to the project. It was court-mandated, and the extractive local groups that live off of suing the government favored it. The Obama administration was generous with federal stimulus funding, and the designs were rushed in order to use stimulus funding to pay for the project’s design, which would be done by consultants rather than with an expansion of in-house hiring. It’s in this atmosphere of profligacy that the project’s cost exploded to, in today’s money, around $3.5 billion, for 7 km of light rail in existing rights-of-way (albeit ones requiring overpass reconstruction).
The project did fall under political threat, after Charlie Baker (R) won the 2014 election. Baker’s impact on Massachusetts governance is fascinating, in that he unambiguously cut its budget significantly in the short run, but also had both before (as budget director in the 1990s) and during (through his actions as governor) wrecked the state’s long-term ability to execute infrastructure, setting up a machine intended to privatize the state and avoiding any in-house hiring. Nonetheless, the direct impact of precarity on GLX was to reduce scope: the betterments were removed and the stations were changed from too big to too small. The final cost was $2.3 billion, or around $2.8 billion in 2022 dollars, and half of that was sunk cot from the previous iteration.
There was no expectation that the project would be canceled – indeed, it was not. A Republican victory was unexpected in a state this left-wing. Then, as Baker was taking office, past governors from both parties expressed optimism that he would get not just GLX done but also the much more complex North-South Rail Link tunnel. Nor did contractors have their contracts yanked unexpectedly, which would get them to bid higher for risk compensation. Baker cold-shouldered not jut NSRL but also much smaller investments in commuter rail, but at no point was anyone stiffed in the process. No: the Patrick-era project was just poorly managed.
 Project selection 
As one caveat, I want to point out a place where precarity does lead to poor project cost-effectiveness: the project selection stage, in the context of tax referendums, especially in Southern California. None of this leads to high per-kilometer costs – Los Angeles’s are exactly as bad as in the rest of the United States – but it does lead to poor project selection, as an unintended consequence of anti-tax New Right politics.
The California-specific issue is that raising taxes requires a referendum, with a two-thirds vote. Swiss referendums are by simple majority, or at most double majority – but in practice most referendums that have a popular vote majority, even a small one, also have a double majority when needed, and in the last 15 years I believe the only two times they didn’t had the double majority veto overturning a 1.5% margin and a 9% margin.
California’s requirement of a two-thirds majority was intended to stop wasteful spending and taxation, but has had the opposite effect. In and around San Francisco, the voters are sufficiently left-wing that two-thirds majorities for social policy are not hard to obtain. But in Southern California, they are not; to build public transportation infrastructure, Los Angeles and San Diego County governments cannot rely on ideology, but instead cut deals with local, non-ideological actors through promising them a piece of the package. Los Angeles measures for transit expansion are, by share of money committed, largely not about transit expansion but operations, new buses, or leakage to roads and bridges; then, what does go to transit expansion is divvied by region, with each region getting something, no matter how cost-ineffective, while core improvements are neglected and so are cross-regional connections (since the local extractive actors aren’t going to ride the trains and can’t tell a circumferential project is useful for them).
This is not a US-wide phenomenon. It’s not even California-wide: this problem is absent from the Bay Area, where BART decided against an expansion to Livermore that was unpopular among technically-oriented advocates, and would like to build more core capacity if it could do so for much less than a billion dollars per km. New York does not have it at all (for one, it doesn’t require two-thirds majorities for budgeting).
At intercity scale, this precarity does cause Amtrak to maximize how many states and congressional districts it runs money-losing daily trains in. But wasting money on night trains and on peripheral regions is hardly a US-only problem – Japan National Railways did that until well past privatization, when its successors spun off money-losing branch lines to prefecture-subsidized third sector railways. This is not at all why there is no plan for Northeastern intercity rail that is worth its weight in dirt: Northeastern rail improvements have been amply funded relative to objective need (if not relative to American costs), and solid investments in the core coexist with wastes of money on the periphery of the network in many countries.
 The issue of politicization 
The precarious, low-cost examples all had to cut costs because of fiscal pressure. However, in all three, the pressure did not include any politicization of engineering questions. Sweden was setting up a civil service modeled on the American Bureau of Public Roads, currently the Federal Highway Administration, which in the middle of the 20th century was a model of depoliticized governance. France and Switzerland have strong civil service bureaucracies – if anything SNCF is too self-contained and needs reorganization, just not if it’s led by the usual French elites or by people from the airline industry.
Importantly, low-cost countries with more clientelism and politicization of the state tend to be more deferential to the expertise of engineers. Greece has a far worse problem of overreliance on political appointees than the United States, let alone the other European democracies; but engineering is somewhat of an exception. Hispanic and Portuguese-speaking cultures put great prestige on engineering, reducing the extent of political micromanagement, even in countries without strong apolitical civil service bureaucracies. Even in Turkey, the politicization of public transportation is entirely at macro level: AKP promises to prioritize investment in areas that vote for it and has denied financing to the Istanbul Metro since the city flipped to the opposition (the city instead borrows money from the European Investment Bank), but below that level there is no micromanagement.
The American examples, in contrast, show much more political micromanagement. This is part of the same package as the privatization of state planning in the globalized system; in the United States often there was never the depoliticization that most of the rest of the developed and middle-income world had, but on top of that, the tendency has been to shut down in-house planning departments or radically shrink them and replace them with consultants. The consultants are then supervised by political appointees with no real qualifications to head capital programs, and the remaining civil servants are browbeaten not to disagree with the political appointees’ proclamations.
Those political appointees rarely measure themselves by any criteria of infrastructure utility. Even in New York they and the managers don’t consistently use the system; in Los Angeles, they use it about as often as the executive director of a well-endowed charity eats at a soup kitchen. To them, the cost is itself a measure of success – and this is true of other agencies, which treat obtaining other people’s money as a mark of achievement and as testament to their power. This behavior then cascades to local advocacy groups, which try to push solutions that maximize outside funding and are at bet indifferent and at worst actively hostile to any attempt at efficiency.
Just giving more support to agencies in their current configuration is not going to help. To the contrary, it only confirms that profligacy gets rewarded. A program of depoliticization of the state is required in tandem with expanding in-house hiring and reversing the globalized system, and the political appointees and the managers and political advocates who are used to dealing with them don’t belong in this program.
	
	
	
	
	2023-01-07
	
	
	Edge Cities With and Without Historic Cores	
	
	
	
	
An edge city is a dense, auto-oriented job center arising from nearby suburban areas, usually without top-down planning. The office parks of Silicon Valley are one such example: the area had a surplus of land and gradually became the core of the American tech industry. In American urbanism, Tysons in Virginia is a common archetype: the area was a minor crossroads until the Capital Beltway made it unusually accessible by car, providing extensive auto-oriented density with little historic core.
But there’s a peculiarity, I think mainly in the suburbs of New York. Unlike archetypal edge cities like Silicon Valley, Tysons, Century City in Los Angeles, or Route 128 north of Boston, some of the edge cities of New York are based on historic cores. Those include White Plains and Stamford, which have had booms in high-end jobs in the last 50 years due to job sprawl, but also Mineola, Tarrytown, and even New Brunswick and Morristown.
The upshot is that it’s much easier to connect these edge cities to public transportation than is typical. In Boston, I’ve spent a lot of time trying to figure out good last mile connections from commuter rail stations. Getting buses to connect outlying residential areas and shopping centers to town center stations is not too hard, but then Route 128 is completely unviable without some major redesign of its road network: the office parks front the freeway in a way that makes it impossible to run buses except dedicated shuttles from one office park to the station, which could never be frequent enough for all-day service. Tysons is investing enormous effort in sprawl repair, which only works because the Washington Metro could be extended there with multiple stations. Far and away, these edge cities are the most difficult case for transit revival for major employment centers.
And in New York, because so much edge city activity is close to historic cores, this is far easier. Stamford and White Plains already have nontrivial if very small transit usage among their workers, usually reverse-commuters who live in New York and take Metro-North. Mineola could too if the LIRR ran reverse-peak service, but it’s about to start doing so. Tarrytown and Sleepy Hollow could be transit-accessible. The New Jersey edge cities are harder – Edison and Woodbridge have lower job density than Downtown Stamford and Downtown White Plains – but there are some office parks that could be made walkable from the train stations.
I don’t know what the history of this peculiar feature is. White Plains and Mineola are both county seats and accreted jobs based on their status as early urban centers in regions that boomed with suburban sprawl in the middle of the 20th century. Tarrytown happened to be the landfall of the Tappan Zee Bridge. Perhaps this is what let them develop into edge cities even while having a much older urban history than Tysons (a decidedly non-urban crossroads until the Beltway was built), Route 128, or Silicon Valley (where San Jose was a latecomer to the tech industry).
What’s true is that all of these edge cities, while fairly close to train stations, are auto-oriented. They’re transit-adjacent but not transit-oriented, in the following ways:
The high-rise office buildings are within walking distance to the train station, but not with a neat density gradient in which the highest development intensity is nearest the station.
The land use at the stations is parking garages for the use of commuters who drive to the station and use the train as a shuttle from a parking lot to Manhattan, rather than as public transportation the way subway riders do.
The streets are fairly hostile to pedestrians, featuring fast car traffic and difficult crossing, without any of the walkability features that city centers have developed in the last 50 years.
The street changes required are fairly subtle. Let us compare White Plains with Metrotown, both image grabs taken from the same altitude:
   
   
These are both edge cities featuring a train station, big buildings, and wide roads. But in Metrotown, the big buildings are next to the train station, and the flat-looking building to its north is the third-largest shopping mall in Canada. The parking goes behind the buildings, with some lots adjoining Kingsway, which has a frequent trolleybus (line 19) but is secondary as a transportation artery to SkyTrain. Farther away, the residential density remains high, with many high-rises in the typical thin-and-tall style of Vancouver. In contrast, in White Plains, one side of the station is a freeway with low-density residential development behind it, and the other is parking garages with office buildings behind them instead of the reverse.
The work required to fix this situation is not extensive. Parking must be removed and replaced with tall buildings, which can be commercial or residential depending on demand. This can be done as part of a transit-first strategy at the municipal level, but can also be compelled top-down if the city objects, since the MTA (and other Northeastern state agencies) has preemption power over local zoning on land it owns, including parking lots and garages.
On the transit side, the usual reforms for improvements in suburban trains and buses would automatically make this viable: high local frequency, integrated bus-rail timetables (to replace the lost parking), integrated fares, etc. The primary target for such reforms is completely different – it’s urban and inner-suburban rail riders – but the beauty of the S-Bahn or RER concept is that it scales well for extending the same high quality of service to the suburbs.
	
	
	
	
	2022-12-30
	
	
	The Four Quadrants of Cities for Transit Revival	
	
	
	
	
Cities that wish to improve their public transportation access and usage are in a bind. Unless they’re already very transit-oriented, they have not only an entrenched economic elite that drives (for example, small business owners almost universally drive), but also have a physical layout that isn’t easy to retrofit even if there is political consensus for modal shift. Thus, to shift travel away from cars, new interventions are needed. Here, there is a distinction between old and new cities. Old cities usually have cores that can be made transit-oriented relatively easily; new cities have demand for new growth, which can be channeled into transit-oriented development. Thus, usually, in both kinds of cities, a considerably degree of modal shift is in fact possible.
However, it’s perhaps best to treat the features of old and new cities separately. The features of old cities that make transit revival possible, that is the presence of a historic core, and those of new cities, that is demand for future growth, are not in perfect negative correlation. In fact, I’m not sure they consistently have negative correlation at all. So this is really a two-by-two diagram, producing four quadrants of potential transit cities.
 Old cities 
The history of public transportation is one of decline in the second half of the 20th century in places that were already rich then; newly-industrialized countries often have different histories. The upshot is that an old auto-oriented place must have been a sizable city before the decline of mass transit, giving it a large core to work from. This core is typically fairly walkable and dense, so transit revival would start from there.
The most successful examples I know of involve the restoration of historic railroads as modern regional lines. Germany is full of small towns that have done so; Hans-Joachim Zierke has some examples of low-cost restoration of regional lines. Overall, Germany writ large must be viewed as such an example: while German economic growth is healthy, population growth is anemic, and the gradual increase in the modal split for public transportation here must be viewed as more intensive reuse of a historic national rail network, anchored by tens of small city cores.
At the level of a metropolitan area, the best candidates for such a revival are similarly old places; in North America, the best I can think of for this are Philadelphia, Boston, and Chicago. Americans don’t perceive any of the three as especially auto-oriented, but their modal splits are comparable to those of small French cities. But in a way, they show one way forward. If there’s a walkable, transit-oriented core, then it may be attractive for people to live near city center; in those three cities it’s also possible to live farther away and commute by subway, but in smaller ones (say, smaller New England cities), the subway is not available but conversely it’s usually affordable to live within walking distance of the historic city center. This creates a New Left-flavored transit revival in that it begins with the dense city center as a locus of consumption, and only then, as a critical mass of people lives there, as a place that it’s worth building new urban rail to.
 New cities 
Usually, if a city has a lot of recent growth from the era in which it has become taken for granted that mobility is by car, then it should have demand for further growth in the future. This demand can be planned around growth zones with a combination of higher residential density and higher job density near rail corridors. The best time to do transit-oriented development is before auto-oriented development patterns even set in.
There are multiple North American examples of how this works. The best is Vancouver, a metropolitan area that has gone from 560,000 people in the 1951 census to 2.6 million in the 2021 census. Ordinarily, one should expect such a region to be entirely auto-oriented, as most American cities with almost entirely postwar growth are; but in 2016, the last census before corona, it had a 20% work trip modal split, and that was before the Evergreen extension opened.
Vancouver has achieved this by using its strong demand for growth to build a high-rise city center, with office towers in the very center and residential ones ringing it, as well as high-density residential neighborhoods next to the Expo Line stations. The biggest suburbs of Vancouver have followed the same plan: Burnaby built an entirely new city center at Metrotown in conjunction with the Expo Line, and even more auto-oriented Surrey has built up Whalley, at the current outer terminal of the line, as one of its main city centers. Housing growth in the region is rapid; YIMBY advocacy calls for more, but the main focus isn’t on broad development (since this already happens) but on permitting more housing in recalcitrant rich areas, led by the West Side, which will soon have its Broadway extension of the Millennium Line.
Less certain but still interesting examples of the same principle are Calgary, Seattle, and Washington. Calgary, a low-density city, planned its growth around the C-Train, and built a high-rise city center, limiting job sprawl even as residential sprawl is extensive; Seattle and the Virginia-side suburbs of Washington have permitted extensive infill housing and this has helped their urban rail systems achieve high ridership by American standards, Seattle even overtaking Philadelphia’s modal split.
 The four quadrants 
The above contrast of old and new cities misses cities that have positive features of both – or neither. The cities with both positive features have the easiest time improving their public transportation systems, and many have never been truly auto-oriented, such as New York or Berlin, to the point that they’re not the best examples to use for how a more auto-oriented city can redevelop as a transit city.
In North America, the best example of both is San Francisco, which simultaneously is an old city with a high-density core and a place with immense demand for growth fueled by the tech industry. The third-generation tech firms – those founded from the mid-2000s onward (Facebook is in a way the last second-generation firm, which generation began with Apple and Microsoft) – have generally headquartered in the city and not in Silicon Valley. Twitter, Uber, Lyft, Airbnb, Dropbox, and Slack are all in the city, and the traditional central business district has expanded to South of Market to accommodate. This is really a combination of the consumption-oriented old-city model, as growing numbers of employees of older second-generation firms chose to live in the city and reverse-commute to Silicon Valley, and the growth-oriented new-city model. Not for nothing, the narrower metropolitan statistical area of San Francisco (without Silicon Valley) reached a modal split of 17% just before corona, the second highest in the United States, with healthy projections for growth.
But then there is the other quadrant, comprising cities that have neither the positive features of old cities nor those of new cities. To be in this quadrant, a city must not be so old as to have a large historic core or an extensive legacy rail network that can be revived, but also be too poor and stagnant to generate new growth demand. Such a city therefore must have grown in a fairly narrow period of time in the early- to mid-20th century. The best example I can think of is Detroit. The consumption-centric model of old city growth can work even there, but it can’t scale well, since there’s not enough of a core compared with the current extent of the population to build out of.
Pedestrian Observations
https://pedestrianobservations.com/
https://pandasthumb.org/
The Panda’s Thumb
2007-11-19
Matt Young
<div>
 
<header>
 <div>
 
 <nav>
 <ul>
<li><a href="/"> Main</a></li>
<li><a href="/archives/index.html"> Archives</a></li>
<li><a href="/search.html"> Search</a></li>
</ul>
 </nav>
 </div>
</header>
 <section>
 <div>
 <article>
 <header>
 <h2><a href="/archives/2023/02/ark-attendance-week.html">Ark attendance weak, at best</a></h2>
 
 
 </header>
 <figure>
<figcaption> Actual Ark attendance vs. projected, based on the data of William Trollinger. The actual attendance in Year 1 seems to be an estimate. Actual attendance for the period March 2020 through February 2021 have been omitted owing to Covid. Actual attendance for the period January through June 2023 is an estimate based on the historical ratio of attendance during the first six months of the year to that of the last six months.</figcaption>
</figure>
<p>William Trollinger, on the blog Righting America, presents what he calls a few facts about Ark Encounter attendance and deems that attendance to be <a href="https://rightingamerica.net/not-even-close-to-what-was-projected-a-few-facts-about-ark-encounter-attendance/">Not Even Close to What Was Projected</a>. Dr. Trollinger bases his conclusion on data acquired by the indomitable Dan Phelps, an occasional guest contributor here at PT. Mr. Phelps each month uses the Kentucky Open Records Act to ascertain the monies collected by the city of Williamstown from the safety fee of $0.50 added to each ticket. From these data it is easy to deduce the number of tickets sold and therefore the number of patrons in any given month, apart from a number of possibly fictitious season-pass holders who pass the ticket window without being assessed the safety fee.</p>
<p>I have graphed Dr. Trollinger’s data and show them as the frontispiece to this article. Two of the years, as noted, are estimates. Year 4 is actually a two-year period from July 2019 to June 2021, but with March 2020 through February 2021 omitted because of the Covid pandemic. The result does not seem out of line. Overall attendance, over the 6 years, is about 40 % of projected attendance and appears to have fallen somewhat since Year 3, as projected attendance rises.</p>
<p>For that, as Dr. Trollinger notes, Williamstown issued $62 million in junk bonds and in effect forwent a significant sum in property taxes. For more details, see the original article.</p>
 
 </article>
 </div>
 <div>
 <article>
 <header>
 <h2><a href="/archives/2023/01/Cryptozoon-occidentale.html">Cryptozoon occidentale</a></h2>
 <p> By Matt Young</p>
 
 </header>
 <p>Photograph by <strong>James Kocher</strong>.</p>
<p>Photography Contest, <strong>Honorable Mention</strong>.</p>
<figure>
<figcaption><i>Cryptozoon occidentale</i> – <a href="https://en.wikipedia.org/wiki/Stromatolite">stromatolite</a> colony, Snowslip Formation/Missoula Group from Essex, Montana. Specimen is a vertical section cut on lapidary saw (i.e., perpendicular to bedding). Age = ~1.0 Ga; neoproterozoic. Width of picture is ~6 cm. See also <a href="https://pubs.er.usgs.gov/publication/pp294D">here</a>.</figcaption>
</figure>
 
 </article>
 </div>
 <div>
 <article>
 <header>
 <h2><a href="/archives/2023/01/cephalopods-better-designed.html">Are cephalopods better designed than humans?</a></h2>
 <p> By Matt Young</p>
 
 </header>
 <figure>
<figcaption>Amur leopard (not a cephalopod), showing retroreflection from tapetum. <small>Colchester Zoo, Monday, November 19, 2007. <a href="https://commons.wikimedia.org/wiki/File:Amur_leopard_by_Keven_Law.jpg">Photograph</a> by Keven Law. Wikimedia Commons, CC BY-SA 2.0.</small>
</figcaption>
</figure>
<p>The answer to the titular question depends on what you think they are designed for; hence, it seems to be a silly question. When it comes to the eyes, however, there is little doubt: human (and other vertebrate) retinas are in backward in that nerves, blood vessels, and other structures are piled on top of the photoreceptors and thus reduce sensitivity, if not resolution. Additionally, the nerves go out through a hole in the retina, which makes us susceptible to glaucoma. Cephalopod retinas, by contrast, face forward, with nothing covering the photoreceptors, and the nerves do not have to pass through a hole in the retina. Cephalopod eyes are thus clearly better designed than human eyes.</p>
<p>I raise this question by way of introducing an article, <a href="https://skepticalinquirer.org/2022/12/the-night-begins-to-shine-the-tapetum-lucidum-and-our-backward-retinas/">The night begins to shine: The tapetum lucidum and our backward retinas</a>, by Nathan H. Lents, in the January-February issue of <i>Skeptical Inquirer</i>. The tapetum lucidum, or just tapetum, is a reflective membrane in the retinas of many nocturnal vertebrates and is located just behind the photoreceptors. Its purpose is presumedly to direct the light that the photoreceptors miss back into them and thereby increase the sensitivity. Interestingly, tapetums composed of many different materials appear to have evolved many different times in different creatures. Cephalopods have no tapetum, presumably because their retinas face forward, with the photoreceptors directly facing the light. Prof. Lents explains all this and more in great but easily understandable detail.</p>
 
 <p><a href="/archives/2023/01/cephalopods-better-designed.html#more">Continue Reading</a> </p>
 
 </article>
 </div>
 <div>
 <article>
 <header>
 <h2><a href="/archives/2023/01/ken-ham-using.html">Ken Ham Using Creation Museum for Kentucky Politics</a></h2>
 <p> By Dan Phelps</p>
 
 </header>
 <figure>
<figcaption>
</figcaption>
</figure>
<p><i><strong>Update</strong>, January 28, 2023, 3:30 p.m., MST. A version of this post appeared in the Lexington (Kentucky) Herald-Leader under the head <a href="https://www.kentucky.com/opinion/op-ed/article271570602.html">Creation Museum to host anti-abortion event with extremist pastor, Ky legislators | Opinion</a>. Mark Looy replied under the head <a href="https://www.kentucky.com/opinion/letters-to-the-editor/article271724962.html">Abortion clarification</a>. You may see the text of his letter in <a href="https://pandasthumb.org/archives/2023/01/ken-ham-using.html#comment-6101095162">Comment 6101095162</a>. Mr. Phelps replies there that Mr. Looy’s reply “seems to concede what I said, but tries to obfuscate. He admits that the legislation would cause women that have abortions be charged with murder and that Pastor Jeff Durbin advocates the death penalty.”</i></p>
<p>Ken Ham and Answers in Genesis (AiG), along with <a href="https://metro.co.uk/2017/02/27/hipster-says-he-wants-to-execute-women-who-have-abortions-6475717/">the extremist Jeff Durbin</a> of Arizona’s Apologia Church, are hosting a political event for pastors at the Creation Museum to support <a href="https://www.youtube.com/watch?v=5lyiv2zQr6k">pending Kentucky anti-abortion legislation</a>. To my knowledge <a href="https://apps.legislature.ky.gov/record/23rs/index_headings.html">this legislation has not yet been introduced</a> as of January 18. The rally is also advertised on January 18, AiG’s weekly “Answers News”, near the end of the show (<a href="https://youtu.be/AUkCdwY6LVs">about 34 minutes in</a>).</p>
<p>Back in March 2007, when the Creation Museum was about to open, AiG’s Mark Looy and Ken Ham penned a post to their website titled “<a href="https://answersingenesis.org/world-religions/humanism/goose-stepping-to-zion/">Goose-stepping to Zion</a>.” This essay claims AiG is “apolitical” and states:</p>
 
 <p><a href="/archives/2023/01/ken-ham-using.html#more">Continue Reading</a> </p>
 
 </article>
 </div>
 <div>
 <article>
 <header>
 <h2><a href="/archives/2023/01/Trioceros-jacksonii.html">Trioceros jacksonii</a></h2>
 <p> By Matt Young</p>
 
 </header>
 <p>Photograph by <strong>Mark Sturtevant</strong>.</p>
<p>Photogrphy Contest, <strong>Honorable Mention</strong>.</p>
<figure>
<figcaption><i>Trioceros jacksonii</i> – Jackson’s chameleon. Dr. Sturtevant writes, "The Hawaiian islands harbor many unique species of plants and animals that have evolved there while in isolation. But of course humans have introduced many non-native species to the islands, and these foreigners come from around the world. This large chameleon was found along a forested path on Maui, and is one of many species of chameleons that thrive on the islands after having been introduced there through the pet trade and general human stupidity. This large chameleon is quite capable of swallowing baby birds that are in the nest, and so it is a threat to the native birds."
</figcaption>
</figure>
 
 </article>
 </div>
</section>
 
 
||||I|||| * Main
 * Archives
 * Search

 Ark attendance weak, at best

 By Matt Young
 February 1, 2023 16:20 MST 0 Comments

 Actual Ark attendance vs. projected, based on the data of William Trollinger. The actual attendance in Year 1 seems to be an estimate. Actual attendance for the period March 2020 through February 2021 have been omitted owing to Covid. Actual attendance for the period January through June 2023 is an estimate based on the historical ratio of attendance during the first six months of the year to that of the last six months.

 William Trollinger, on the blog Righting America, presents what he calls a few facts about Ark Encounter attendance and deems that attendance to be Not Even Close to What Was Projected. Dr. Trollinger bases his conclusion on data acquired by the indomitable Dan Phelps, an occasional guest contributor here at PT. Mr. Phelps each month uses the Kentucky Open Records Act to ascertain the monies collected by the city of Williamstown from the safety fee of $0.50 added to each ticket. From these data it is easy to deduce the number of tickets sold and therefore the number of patrons in any given month, apart from a number of possibly fictitious season-pass holders who pass the ticket window without being assessed the safety fee.

 I have graphed Dr. Trollinger’s data and show them as the frontispiece to this article. Two of the years, as noted, are estimates. Year 4 is actually a two-year period from July 2019 to June 2021, but with March 2020 through February 2021 omitted because of the Covid pandemic. The result does not seem out of line. Overall attendance, over the 6 years, is about 40 % of projected attendance and appears to have fallen somewhat since Year 3, as projected attendance rises.

 For that, as Dr. Trollinger notes, Williamstown issued $62 million in junk bonds and in effect forwent a significant sum in property taxes. For more details, see the original article.

 Cryptozoon occidentale

 By Matt Young
 January 30, 2023 12:00 MST 0 Comments

 Photograph by James Kocher.

 Photography Contest, Honorable Mention.

 Cryptozoon occidentale – stromatolite colony, Snowslip Formation/Missoula Group from Essex, Montana. Specimen is a vertical section cut on lapidary saw (i.e., perpendicular to bedding). Age = ~1.0 Ga; neoproterozoic. Width of picture is ~6 cm. See also here.

 Are cephalopods better designed than humans?

 By Matt Young
 January 23, 2023 08:42 MST 0 Comments

 Amur leopard (not a cephalopod), showing retroreflection from tapetum. Colchester Zoo, Monday, November 19, 2007. Photograph by Keven Law. Wikimedia Commons, CC BY-SA 2.0.

 The answer to the titular question depends on what you think they are designed for; hence, it seems to be a silly question. When it comes to the eyes, however, there is little doubt: human (and other vertebrate) retinas are in backward in that nerves, blood vessels, and other structures are piled on top of the photoreceptors and thus reduce sensitivity, if not resolution. Additionally, the nerves go out through a hole in the retina, which makes us susceptible to glaucoma. Cephalopod retinas, by contrast, face forward, with nothing covering the photoreceptors, and the nerves do not have to pass through a hole in the retina. Cephalopod eyes are thus clearly better designed than human eyes.

 I raise this question by way of introducing an article, The night begins to shine: The tapetum lucidum and our backward retinas, by Nathan H. Lents, in the January-February issue of Skeptical Inquirer. The tapetum lucidum, or just tapetum, is a reflective membrane in the retinas of many nocturnal vertebrates and is located just behind the photoreceptors. Its purpose is presumedly to direct the light that the photoreceptors miss back into them and thereby increase the sensitivity. Interestingly, tapetums composed of many different materials appear to have evolved many different times in different creatures. Cephalopods have no tapetum, presumably because their retinas face forward, with the photoreceptors directly facing the light. Prof. Lents explains all this and more in great but easily understandable detail.

 Continue Reading

 Ken Ham Using Creation Museum for Kentucky Politics

 By Dan Phelps
 January 20, 2023 09:45 MST 0 Comments

 Update, January 28, 2023, 3:30 p.m., MST. A version of this post appeared in the Lexington (Kentucky) Herald-Leader under the head Creation Museum to host anti-abortion event with extremist pastor, Ky legislators | Opinion. Mark Looy replied under the head Abortion clarification. You may see the text of his letter in Comment 6101095162. Mr. Phelps replies there that Mr. Looy’s reply “seems to concede what I said, but tries to obfuscate. He admits that the legislation would cause women that have abortions be charged with murder and that Pastor Jeff Durbin advocates the death penalty.”

 Ken Ham and Answers in Genesis (AiG), along with the extremist Jeff Durbin of Arizona’s Apologia Church, are hosting a political event for pastors at the Creation Museum to support pending Kentucky anti-abortion legislation. To my knowledge this legislation has not yet been introduced as of January 18. The rally is also advertised on January 18, AiG’s weekly “Answers News”, near the end of the show (about 34 minutes in).

 Back in March 2007, when the Creation Museum was about to open, AiG’s Mark Looy and Ken Ham penned a post to their website titled “Goose-stepping to Zion.” This essay claims AiG is “apolitical” and states:

 Continue Reading

 Trioceros jacksonii

 By Matt Young
 January 16, 2023 12:00 MST 0 Comments

 Photograph by Mark Sturtevant.

 Photogrphy Contest, Honorable Mention.

 Trioceros jacksonii – Jackson’s chameleon. Dr. Sturtevant writes, "The Hawaiian islands harbor many unique species of plants and animals that have evolved there while in isolation. But of course humans have introduced many non-native species to the islands, and these foreigners come from around the world. This large chameleon was found along a forested path on Maui, and is one of many species of chameleons that thrive on the islands after having been introduced there through the pet trade and general human stupidity. This large chameleon is quite capable of swallowing baby birds that are in the nest, and so it is a threat to the native birds."

 To see earlier posts, select the Archives at the top of this page    
     
 Recent Comments     
     
 To see the comment in context of the discussion click on the text that indicates how long ago the comment was posted, such as "2 hours ago". Then wait for the post and then the comments to load (may take many seconds). The comment should have a gray vertical bar to the left of the commenter's avatar.
     
 Copyright © The Panda’s Thumb and original authors — Content provided under Creative Commons BY-NC-ND License 4.0.   
     
 
Main 
  Archives 
  Search 
 
 
 
 
 
 
  Ark attendance weak, at best 
 
 
 
 
 Actual Ark attendance vs. projected, based on the data of William Trollinger. The actual attendance in Year 1 seems to be an estimate. Actual attendance for the period March 2020 through February 2021 have been omitted owing to Covid. Actual attendance for the period January through June 2023 is an estimate based on the historical ratio of attendance during the first six months of the year to that of the last six months.
William Trollinger, on the blog Righting America, presents what he calls a few facts about Ark Encounter attendance and deems that attendance to be Not Even Close to What Was Projected. Dr. Trollinger bases his conclusion on data acquired by the indomitable Dan Phelps, an occasional guest contributor here at PT. Mr. Phelps each month uses the Kentucky Open Records Act to ascertain the monies collected by the city of Williamstown from the safety fee of $0.50 added to each ticket. From these data it is easy to deduce the number of tickets sold and therefore the number of patrons in any given month, apart from a number of possibly fictitious season-pass holders who pass the ticket window without being assessed the safety fee.
I have graphed Dr. Trollinger’s data and show them as the frontispiece to this article. Two of the years, as noted, are estimates. Year 4 is actually a two-year period from July 2019 to June 2021, but with March 2020 through February 2021 omitted because of the Covid pandemic. The result does not seem out of line. Overall attendance, over the 6 years, is about 40 % of projected attendance and appears to have fallen somewhat since Year 3, as projected attendance rises.
For that, as Dr. Trollinger notes, Williamstown issued $62 million in junk bonds and in effect forwent a significant sum in property taxes. For more details, see the original article.
 
 
 
 
 
 
  Cryptozoon occidentale 
  By Matt Young
 
 
 Photograph by James Kocher.
Photography Contest, Honorable Mention.
 Cryptozoon occidentale – stromatolite colony, Snowslip Formation/Missoula Group from Essex, Montana. Specimen is a vertical section cut on lapidary saw (i.e., perpendicular to bedding). Age = ~1.0 Ga; neoproterozoic. Width of picture is ~6 cm. See also here.
 
 
 
 
 
 
  Are cephalopods better designed than humans? 
  By Matt Young
 
 
 
Amur leopard (not a cephalopod), showing retroreflection from tapetum. Colchester Zoo, Monday, November 19, 2007. Photograph by Keven Law. Wikimedia Commons, CC BY-SA 2.0.
The answer to the titular question depends on what you think they are designed for; hence, it seems to be a silly question. When it comes to the eyes, however, there is little doubt: human (and other vertebrate) retinas are in backward in that nerves, blood vessels, and other structures are piled on top of the photoreceptors and thus reduce sensitivity, if not resolution. Additionally, the nerves go out through a hole in the retina, which makes us susceptible to glaucoma. Cephalopod retinas, by contrast, face forward, with nothing covering the photoreceptors, and the nerves do not have to pass through a hole in the retina. Cephalopod eyes are thus clearly better designed than human eyes.
I raise this question by way of introducing an article, The night begins to shine: The tapetum lucidum and our backward retinas, by Nathan H. Lents, in the January-February issue of Skeptical Inquirer. The tapetum lucidum, or just tapetum, is a reflective membrane in the retinas of many nocturnal vertebrates and is located just behind the photoreceptors. Its purpose is presumedly to direct the light that the photoreceptors miss back into them and thereby increase the sensitivity. Interestingly, tapetums composed of many different materials appear to have evolved many different times in different creatures. Cephalopods have no tapetum, presumably because their retinas face forward, with the photoreceptors directly facing the light. Prof. Lents explains all this and more in great but easily understandable detail.
 
  Continue Reading 
 
 
 
 
 
 
  Ken Ham Using Creation Museum for Kentucky Politics 
  By Dan Phelps
 
 
 
  Update, January 28, 2023, 3:30 p.m., MST. A version of this post appeared in the Lexington (Kentucky) Herald-Leader under the head Creation Museum to host anti-abortion event with extremist pastor, Ky legislators | Opinion. Mark Looy replied under the head Abortion clarification. You may see the text of his letter in Comment 6101095162. Mr. Phelps replies there that Mr. Looy’s reply “seems to concede what I said, but tries to obfuscate. He admits that the legislation would cause women that have abortions be charged with murder and that Pastor Jeff Durbin advocates the death penalty.” 
Ken Ham and Answers in Genesis (AiG), along with the extremist Jeff Durbin of Arizona’s Apologia Church, are hosting a political event for pastors at the Creation Museum to support pending Kentucky anti-abortion legislation. To my knowledge this legislation has not yet been introduced as of January 18. The rally is also advertised on January 18, AiG’s weekly “Answers News”, near the end of the show (about 34 minutes in).
Back in March 2007, when the Creation Museum was about to open, AiG’s Mark Looy and Ken Ham penned a post to their website titled “Goose-stepping to Zion.” This essay claims AiG is “apolitical” and states:
 
  Continue Reading 
 
 
 
 
 
 
  Trioceros jacksonii 
  By Matt Young
 
 
 Photograph by Mark Sturtevant.
Photogrphy Contest, Honorable Mention.
 Trioceros jacksonii – Jackson’s chameleon. Dr. Sturtevant writes, "The Hawaiian islands harbor many unique species of plants and animals that have evolved there while in isolation. But of course humans have introduced many non-native species to the islands, and these foreigners come from around the world. This large chameleon was found along a forested path on Maui, and is one of many species of chameleons that thrive on the islands after having been introduced there through the pet trade and general human stupidity. This large chameleon is quite capable of swallowing baby birds that are in the nest, and so it is a threat to the native birds."
The Panda’s Thumb
http://pandasthumb.org/
https://blogs.scientificamerican.com/
Home
2020-04-30
None
<div><div><h2>Recent Posts</h2><p><a href="#topic-list">Select Topic </a></p><ul><li><a href="?category=biology">Biology</a></li><li><a href="?category=chemistry">Chemistry</a></li><li><a href="?category=environment">Environment</a></li><li><a href="?category=health">Health</a></li><li><a href="?category=math">Math</a></li><li><a href="?category=mind-and-brain">Mind &amp; Brain</a></li><li><a href="?category=social-sciences">Social Sciences</a></li><li><a href="?category=space-and-physics">Space &amp; Physics</a></li><li><a href="?category=technology">Technology</a></li><li><a href="?category=the-sciences">The Sciences</a></li></ul><div><article><a href="https://blogs.scientificamerican.com/dog-spies/good-and-bad-ways-to-help-a-dog-afraid-of-fireworks/"><div><p>Dog Spies</p></div><div><h2>Good (and Bad) Ways to Help a Dog Afraid of Fireworks</h2><p>You are more important than you think </p><p>July 2, 2022 — Julie Hecht | Opinion</p></div></a></article><article><a href="https://blogs.scientificamerican.com/voices/one-more-pioneering-woman-in-science-youve-probably-never-heard-of/"><div><p>Voices</p></div><div><h2>One More Pioneering Woman in Science You’ve Probably Never Heard of</h2><p>Elizabeth Stern’s research led the way to our modern understanding of the prevention, treatment and diagnosis of cervical cancer</p><p>March 11, 2022 — Ellen Elliott | Opinion</p></div></a></article><article><a href="https://blogs.scientificamerican.com/guest-blog/3-revolutionary-women-of-mathematics/"><div><p>Guest Blog</p></div><div><h2>3 Revolutionary Women of Mathematics</h2><p>Everyone knows that history's great mathematicians were all men—but everybody is wrong</p><p>March 9, 2022 — Avery Carr</p></div></a></article><article><a href="https://blogs.scientificamerican.com/cross-check/thanksgiving-and-the-myth-of-native-american-savages/"><div><p>Cross-Check</p></div><div><h2>Thanksgiving and the Myth of Native American ‘Savages’</h2><p>Prominent scientists exaggerate the violence of Native Americans, whom European invaders ravaged</p><p>November 24, 2021 — John Horgan | Opinion</p></div></a></article><article><a href="https://blogs.scientificamerican.com/anthropology-in-practice/the-rise-of-pumpkin-spice/"><div><p>Anthropology in Practice</p></div><div><h2>The Rise of Pumpkin Spice</h2><p>Pumpkins hold no special status for Americans unless it’s fall. Where did the obsession with pumpkin flavors come from?</p><p>November 23, 2021 — Krystal D'Costa | Opinion</p></div></a></article><article><a href="https://blogs.scientificamerican.com/observations/time-has-no-meaning-at-the-north-pole/"><div><p>Observations</p></div><div><h2>Time Has No Meaning at the North Pole</h2><p>The utter lack of time zones, daylight and people creates a bizarre world</p><p>November 7, 2021 — Katie Weeman | Opinion</p></div></a></article><article><a href="https://blogs.scientificamerican.com/voices/george-floyds-autopsy-and-the-structural-gaslighting-of-america/"><div><p>Voices</p></div><div><h2>George Floyd’s Autopsy and the Structural Gaslighting of America</h2><p>The weaponization of medical language emboldened white supremacy with the authority of the white coat. How will we stop it from happening again?</p><p>March 29, 2021 — Ann Crawford-Roberts, Sonya Shadravan, Jennifer Tsai, Nicolás E. Barceló, Allie Gips, Michael Mensah, Nichole Roxas, Alina Kung, Anna Darby, Naya Misa, Isabella Morton and Alice Shen | Opinion</p></div></a></article><article><a href="https://blogs.scientificamerican.com/observations/green-alternative-to-border-wall-might-have-saved-texas/"><div><p>Observations</p></div><div><h2>Green Alternative to Border Wall Might Have Saved Texas</h2><p>A U.S.-Mexico corridor of renewable energy and water could have prevented widespread emergencies</p><p>March 4, 2021 — Mark Fischetti | Opinion</p></div></a></article><article><a href="https://blogs.scientificamerican.com/observations/a-half-century-later-the-journey-of-apollo-8-still-inspires/"><div><p>Observations</p></div><div><h2>A Half Century Later, the Journey of <i>Apollo 8</i> Still Inspires</h2><p>On Christmas Eve 1968 people worldwide stepped back from the throes of crises and recognized themselves as citizens of a tiny, fragile planet</p><p>December 24, 2020 — Michael J. Battaglia | Opinion</p></div></a></article><article><a href="https://blogs.scientificamerican.com/plugged-in/dont-throw-away-those-leftovers-saving-food-will-save-energy/"><div><p>Plugged In</p></div><div><h2>Don't Throw Away Those Leftovers! Saving Food Will Save Energy</h2><p>Don’t be too quick to throw away those Thanksgiving leftovers. By saving that turkey and stuffing to eat another day, you can help save a lot of energy from ending up in the garbage bin...</p><p>November 27, 2020 — Melissa C. Lott | Opinion</p></div></a></article></div><h2>Blog Index</h2><div><ul>	<li><section><div><article><p></p><h3>@ScientificAmerican</h3><p>STAFFBehind the scenes at Scientific American</p><a href="https://blogs.scientificamerican.com/at-scientific-american/">Read</a></article></div><div><article><p></p><h3>Anecdotes from the Archive</h3><p>STAFF</p><a href="https://blogs.scientificamerican.com/anecdotes-from-the-archive/">Read</a></article></div><div><article><p></p><h3>Anthropology in Practice</h3><p>Exploring the human condition.</p><a href="https://blogs.scientificamerican.com/anthropology-in-practice/">Read</a></article></div><div><article><p></p><h3>Beautiful Minds</h3><p>Insights into intelligence, creativity, personality, and well-being</p><a href="https://blogs.scientificamerican.com/beautiful-minds/">Read</a></article></div><div><article><p></p><h3>Budding Scientist</h3><p>Everything you always wanted to know about raising science-literate kids</p><a href="https://blogs.scientificamerican.com/budding-scientist/">Read</a></article></div><div><article><p></p><h3>Cross-Check</h3><p>Critical views of science in the news</p><a href="https://blogs.scientificamerican.com/cross-check/">Read</a></article></div><div><article><p></p><h3>Dark Star Diaries</h3><p>STAFF</p><a href="https://blogs.scientificamerican.com/dark-star-diaries/">Read</a></article></div><div><article><p></p><h3>Dog Spies</h3><p>Explore the science behind the dog in your bed</p><a href="https://blogs.scientificamerican.com/dog-spies/">Read</a></article></div></section></li>	<li><section><div><article><p></p><h3>Extinction Countdown</h3><p>News and research about endangered species from around the world</p><a href="https://blogs.scientificamerican.com/extinction-countdown/">Read</a></article></div><div><article><p></p><h3>Eye of the Storm</h3><p>The Science Behind Extreme Weather</p><a href="https://blogs.scientificamerican.com/eye-of-the-storm/">Read</a></article></div><div><article><p></p><h3>Frontiers for Young Minds</h3><p>Science by and for kids ages 8-15</p><a href="https://blogs.scientificamerican.com/frontiers-for-young-minds/">Read</a></article></div><div><article><p></p><h3>Guest Blog</h3><p>Commentary invited by editors of Scientific American</p><a href="https://blogs.scientificamerican.com/guest-blog/">Read</a></article></div><div><article><p></p><h3>Hot Planet</h3><p>Climate science in a changing world</p><a href="https://blogs.scientificamerican.com/hot-planet/">Read</a></article></div><div><article><p></p><h3>Illusion Chasers</h3><p>Illusions, Delusions, and Everyday Deceptions</p><a href="https://blogs.scientificamerican.com/illusion-chasers/">Read</a></article></div><div><article><p></p><h3>Life, Unbounded</h3><p>Discussion and news about planets, exoplanets, and astrobiology</p><a href="https://blogs.scientificamerican.com/life-unbounded/">Read</a></article></div><div><article><p></p><h3>Observations</h3><p>Opinion, arguments &amp; analyses from guest experts and from the editors of <em>Scientific American</em></p><a href="https://blogs.scientificamerican.com/observations/">Read</a></article></div></section></li>	<li><section><div><article><p></p><h3>Roots of Unity</h3><p>Mathematics: learning it, doing it, celebrating it.</p><a href="https://blogs.scientificamerican.com/roots-of-unity/">Read</a></article></div><div><article><p></p><h3>Rosetta Stones</h3><p>Adventures in the good science of rock-breaking.</p><a href="https://blogs.scientificamerican.com/rosetta-stones/">Read</a></article></div><div><article><p></p><h3>SA Visual</h3><p>STAFFIllustrating science since 1845</p><a href="https://blogs.scientificamerican.com/sa-visual/">Read</a></article></div><div><article><p></p><h3>Talking back</h3><p>STAFFA science blog, <em>sans blague</em></p><a href="https://blogs.scientificamerican.com/talking-back/">Read</a></article></div><div><article><p></p><h3>The Artful Amoeba</h3><p>A Blog About the Weird Wonderfulness of Life on Earth</p><a href="https://blogs.scientificamerican.com/artful-amoeba/">Read</a></article></div><div><article><p></p><h3>Voices</h3><p>Exploring and celebrating diversity in science.</p><a href="https://blogs.scientificamerican.com/voices/">Read</a></article></div></section></li></ul>	<div><ul><li><a href="#">Previous</a></li><li><a href="#">Next</a></li></ul></div></div></div>||||I||||Skip to main content
 SubscribeLatest Issues Menu
 Scientific American
 * Cart 0
 * Sign In Email *:Password *:Forgot password?LoginNot yet registered?
 * |Newsletters
 SearchSubscribe
 Close
 Search InputSearchClose
 SearchAdvanced Search
 * COVID
 * Health
 * Mind & Brain
 * Environment
 * Technology
 * Space & Physics
 * Video
 * Podcasts
 * Opinion
 * Store
 * Subscribe
 * Current Issue


 * Cart 0
 * Sign In
 * Newsletters

 Knowledge is Power

 Shop Now

 Blogs

 * View Blog Index
 Observations

 Masks and Emasculation: Why Some Men Refuse to Take Safety Precautions

 They think it makes them look weak, and avoiding that is evidently more important to them than demonstrating responsible behavior

 * April 30, 2020 — Peter Glick
 Observations

 How a Landmark Physics Paper from the 1970s Uncannily Describes the COVID-19 Pandemic

 Phil Anderson’s article “More Is Different” describes how different levels of complexity require new ways of thinking. And as the virus multiplies and spreads, that’s just what the human race desperately needs...

 * April 29, 2020 — Clifford Brangwynne | Opinion
 Observations

 The Fight against COVID-19 Threatens to Cause Collateral Health Damage

 The pandemic is no excuse to abandon chronic disease management and prevention

 * May 1, 2020 — Rachel Issaka | Opinion

 Recent Posts

 Select Topic
 * Biology
 * Chemistry
 * Environment
 * Health
 * Math
 * Mind & Brain
 * Social Sciences
 * Space & Physics
 * Technology
 * The Sciences
 Dog Spies

 Good (and Bad) Ways to Help a Dog Afraid of Fireworks

 You are more important than you think

 July 2, 2022 — Julie Hecht | Opinion
 Voices

 One More Pioneering Woman in Science You’ve Probably Never Heard of

 Elizabeth Stern’s research led the way to our modern understanding of the prevention, treatment and diagnosis of cervical cancer

 March 11, 2022 — Ellen Elliott | Opinion
 Guest Blog

 3 Revolutionary Women of Mathematics

 Everyone knows that history's great mathematicians were all men—but everybody is wrong

 March 9, 2022 — Avery Carr
 Cross-Check

 Thanksgiving and the Myth of Native American ‘Savages’

 Prominent scientists exaggerate the violence of Native Americans, whom European invaders ravaged

 November 24, 2021 — John Horgan | Opinion
 Anthropology in Practice

 The Rise of Pumpkin Spice

 Pumpkins hold no special status for Americans unless it’s fall. Where did the obsession with pumpkin flavors come from?

 November 23, 2021 — Krystal D'Costa | Opinion
 Observations

 Time Has No Meaning at the North Pole

 The utter lack of time zones, daylight and people creates a bizarre world

 November 7, 2021 — Katie Weeman | Opinion
 Voices

 George Floyd’s Autopsy and the Structural Gaslighting of America

 The weaponization of medical language emboldened white supremacy with the authority of the white coat. How will we stop it from happening again?

 March 29, 2021 — Ann Crawford-Roberts, Sonya Shadravan, Jennifer Tsai, Nicolás E. Barceló, Allie Gips, Michael Mensah, Nichole Roxas, Alina Kung, Anna Darby, Naya Misa, Isabella Morton and Alice Shen | Opinion
 Observations

 Green Alternative to Border Wall Might Have Saved Texas

 A U.S.-Mexico corridor of renewable energy and water could have prevented widespread emergencies

 March 4, 2021 — Mark Fischetti | Opinion
 Observations

 A Half Century Later, the Journey of Apollo 8 Still Inspires

 On Christmas Eve 1968 people worldwide stepped back from the throes of crises and recognized themselves as citizens of a tiny, fragile planet

 December 24, 2020 — Michael J. Battaglia | Opinion
 Plugged In

 Don't Throw Away Those Leftovers! Saving Food Will Save Energy

 Don’t be too quick to throw away those Thanksgiving leftovers. By saving that turkey and stuffing to eat another day, you can help save a lot of energy from ending up in the garbage bin...

 November 27, 2020 — Melissa C. Lott | Opinion

 Advertisement

 1. 1
 2. 2
 3. 2
 4. 3
 5. 4
 6. 5
 7. ...
 8. 10
 9. 20
 10. 30
 Next

 Blog Index

 * @ScientificAmerican

 STAFF Behind the scenes at Scientific American

 Read

 Anecdotes from the Archive

 STAFF

 Read

 Anthropology in Practice

 Exploring the human condition.

 Read

 Beautiful Minds

 Insights into intelligence, creativity, personality, and well-being

 Read

 Budding Scientist

 Everything you always wanted to know about raising science-literate kids

 Read

 Cross-Check

 Critical views of science in the news

 Read

 Dark Star Diaries

 STAFF

 Read

 Dog Spies

 Explore the science behind the dog in your bed

 Read

 * Extinction Countdown

 News and research about endangered species from around the world

 Read

 Eye of the Storm

 The Science Behind Extreme Weather

 Read

 Frontiers for Young Minds

 Science by and for kids ages 8-15

 Read

 Guest Blog

 Commentary invited by editors of Scientific American

 Read

 Hot Planet

 Climate science in a changing world

 Read

 Illusion Chasers

 Illusions, Delusions, and Everyday Deceptions

 Read

 Life, Unbounded

 Discussion and news about planets, exoplanets, and astrobiology

 Read

 Observations

 Opinion, arguments & analyses from guest experts and from the editors of Scientific American

 Read

 * Roots of Unity

 Mathematics: learning it, doing it, celebrating it.

 Read

 Rosetta Stones

 Adventures in the good science of rock-breaking.

 Read

 SA Visual

 STAFF Illustrating science since 1845

 Read

 Talking back

 STAFF A science blog, sans blague

 Read

 The Artful Amoeba

 A Blog About the Weird Wonderfulness of Life on Earth

 Read

 Voices

 Exploring and celebrating diversity in science.

 Read
 * Previous
 * Next

 Support Science Journalism

 Discover world-changing science. Explore our digital archive back to 1845, including articles by more than 150 Nobel Prize winners.

 Subscribe Now!

Follow us

 * instagram
 * soundcloud
 * youtube
 * twitter
 * facebook
 * rss

 Scientific american arabic

 العربية
 * Return & Refund Policy
 * About
 * Press Room
 * FAQs
 * Contact Us
 * Site Map
 * Advertise
 * SA Custom Media
 * Terms of Use
 * Privacy Policy
 * California Consumer Privacy Statement
 * Use of cookies/Do not sell my data
 * International Editions
 Scientific American is part of Springer Nature, which owns or has commercial relations with thousands of scientific publications (many of them can be found at www.springernature.com/us). Scientific American maintains a strict policy of editorial independence in reporting developments in science to our readers.

 © 2023 Scientific American, a Division of Nature America, Inc.

 All Rights Reserved.

Scroll To Top

 Support science journalism.

 Thanks for reading Scientific American. Knowledge awaits.

 See Subscription Options

 Already a subscriber? Sign in.

 Thanks for reading Scientific American. Create your free account or Sign in to continue.

 Create Account

 See Subscription Options

 Continue reading with a Scientific American subscription.

 Subscribe Now

 You may cancel at any time.

 Sign in.
Recent Posts  Select Topic     Biology   Chemistry   Environment   Health   Math   Mind &amp; Brain   Social Sciences   Space &amp; Physics   Technology   The Sciences       Dog Spies   Good (and Bad) Ways to Help a Dog Afraid of Fireworks You are more important than you think  July 2, 2022 — Julie Hecht | Opinion       Voices   One More Pioneering Woman in Science You’ve Probably Never Heard of Elizabeth Stern’s research led the way to our modern understanding of the prevention, treatment and diagnosis of cervical cancer March 11, 2022 — Ellen Elliott | Opinion       Guest Blog   3 Revolutionary Women of Mathematics Everyone knows that history's great mathematicians were all men—but everybody is wrong March 9, 2022 — Avery Carr       Cross-Check   Thanksgiving and the Myth of Native American ‘Savages’ Prominent scientists exaggerate the violence of Native Americans, whom European invaders ravaged November 24, 2021 — John Horgan | Opinion       Anthropology in Practice   The Rise of Pumpkin Spice Pumpkins hold no special status for Americans unless it’s fall. Where did the obsession with pumpkin flavors come from? November 23, 2021 — Krystal D'Costa | Opinion       Observations   Time Has No Meaning at the North Pole The utter lack of time zones, daylight and people creates a bizarre world November 7, 2021 — Katie Weeman | Opinion       Voices   George Floyd’s Autopsy and the Structural Gaslighting of America The weaponization of medical language emboldened white supremacy with the authority of the white coat. How will we stop it from happening again? March 29, 2021 — Ann Crawford-Roberts, Sonya Shadravan, Jennifer Tsai, Nicolás E. Barceló, Allie Gips, Michael Mensah, Nichole Roxas, Alina Kung, Anna Darby, Naya Misa, Isabella Morton and Alice Shen | Opinion       Observations   Green Alternative to Border Wall Might Have Saved Texas A U.S.-Mexico corridor of renewable energy and water could have prevented widespread emergencies March 4, 2021 — Mark Fischetti | Opinion       Observations   A Half Century Later, the Journey of Apollo 8 Still Inspires On Christmas Eve 1968 people worldwide stepped back from the throes of crises and recognized themselves as citizens of a tiny, fragile planet December 24, 2020 — Michael J. Battaglia | Opinion       Plugged In   Don't Throw Away Those Leftovers! Saving Food Will Save Energy Don’t be too quick to throw away those Thanksgiving leftovers. By saving that turkey and stuffing to eat another day, you can help save a lot of energy from ending up in the garbage bin... November 27, 2020 — Melissa C. Lott | Opinion     Blog Index  	      @ScientificAmerican STAFFBehind the scenes at Scientific American Read       Anecdotes from the Archive STAFF Read       Anthropology in Practice Exploring the human condition. Read       Beautiful Minds Insights into intelligence, creativity, personality, and well-being Read       Budding Scientist Everything you always wanted to know about raising science-literate kids Read       Cross-Check Critical views of science in the news Read       Dark Star Diaries STAFF Read       Dog Spies Explore the science behind the dog in your bed Read    	      Extinction Countdown News and research about endangered species from around the world Read       Eye of the Storm The Science Behind Extreme Weather Read       Frontiers for Young Minds Science by and for kids ages 8-15 Read       Guest Blog Commentary invited by editors of Scientific American Read       Hot Planet Climate science in a changing world Read       Illusion Chasers Illusions, Delusions, and Everyday Deceptions Read       Life, Unbounded Discussion and news about planets, exoplanets, and astrobiology Read       Observations Opinion, arguments &amp; analyses from guest experts and from the editors of Scientific American  Read    	      Roots of Unity Mathematics: learning it, doing it, celebrating it. Read       Rosetta Stones Adventures in the good science of rock-breaking. Read       SA Visual STAFFIllustrating science since 1845 Read       Talking back STAFFA science blog, sans blague  Read       The Artful Amoeba A Blog About the Weird Wonderfulness of Life on Earth Read       Voices Exploring and celebrating diversity in science. Read     	   Previous   Next
Home
https://blogs.scientificamerican.com/
https://www.in-mind.org/blog
the InMind blog | In-Mind
2024-04-24
None
<div><div>
<div>
<div>
<div>
<p>
	24.04.2024 | 
	blog categorie(s) - <a href="https://www.in-mind.org/blog/gender">Gender</a> </p>
</div>
<h2><a href="https://www.in-mind.org/blog/post/its-just-a-joke-right-empirical-findings-on-the-serious-effects-of-sexist-humor">It’s just a joke, right? Empirical findings on the serious effects of sexist humor</a></h2>
<p><a href="https://www.in-mind.org/blog/post/its-just-a-joke-right-empirical-findings-on-the-serious-effects-of-sexist-humor"></a> </p>
<div>
<h2><a href="https://www.in-mind.org/blog/post/its-just-a-joke-right-empirical-findings-on-the-serious-effects-of-sexist-humor">It’s just a joke, right? Empirical findings on the serious effects of sexist humor</a></h2>
<p>
	You can still make a joke, right? Yes, but... misogynistic jokes about women can have negative consequences. What can such humor do to women and men?	
</p>
</div>
</div> <div>
<div>
<p>
	22.04.2024 | 
	blog categorie(s) - <a href="https://www.in-mind.org/blog/other">Other</a> </p>
</div>
<h2><a href="https://www.in-mind.org/blog/post/breakfast-is-the-most-important-meal-of-the-dayor-is-it">Breakfast is the most important meal of the day…or is it?</a></h2>
<p><a href="https://www.in-mind.org/blog/post/breakfast-is-the-most-important-meal-of-the-dayor-is-it"></a> </p>
<div>
<h2><a href="https://www.in-mind.org/blog/post/breakfast-is-the-most-important-meal-of-the-dayor-is-it">Breakfast is the most important meal of the day…or is it?</a></h2>
<p>
	Many people follow the motto “breakfast is the most important meal of the day”. On the other hand, intermittent fasting, which often involves deliberately skipping breakfast, is becoming increasingly popular. How does this fit together? What, how much and when you should eat is a much-discussed topic with countless opinions. For example, who hasn't heard the saying: "Breakfast is the most important meal of the day"? Most people would perhaps attribute this to the wisdom and life experience of their...	
</p>
</div>
</div> <div>
<h2><a href="https://www.in-mind.org/blog/post/how-harmful-is-always-on-for-our-well-being-technology-assisted-supplemental-work">How harmful is “always-on” for our well-being? Technology-assisted supplemental work</a></h2>
<p><a href="https://www.in-mind.org/blog/post/how-harmful-is-always-on-for-our-well-being-technology-assisted-supplemental-work"></a> </p>
<div>
<h2><a href="https://www.in-mind.org/blog/post/how-harmful-is-always-on-for-our-well-being-technology-assisted-supplemental-work">How harmful is “always-on” for our well-being? Technology-assisted supplemental work</a></h2>
<p>
	Calling a colleague on the way home, finishing a presentation after dinner or checking emails before going to bed - for many employees, this is the norm rather than an exception. But to what extent does being "always on" jeopardize our well-being? And what can help us to benefit from additional work after-hours? For most employees, smartphones, laptops, and co are an integral part of their working lives. In many professions, information and communication technologies such as E-Mail and messenger...	
</p>
</div>
</div> <div>
<h2><a href="https://www.in-mind.org/blog/post/olympic-mind-an-international-special-issue-on-the-psychology-of-extraordinary-sports">Olympic Mind: An International Special Issue on The Psychology of Extraordinary Sports Performance</a></h2>
<p><a href="https://www.in-mind.org/blog/post/olympic-mind-an-international-special-issue-on-the-psychology-of-extraordinary-sports"></a> </p>
<div>
<h2><a href="https://www.in-mind.org/blog/post/olympic-mind-an-international-special-issue-on-the-psychology-of-extraordinary-sports">Olympic Mind: An International Special Issue on The Psychology of Extraordinary Sports Performance</a></h2>
<p>
	Performing the Biles 3 in gymnastics, running 100m in 9.58 seconds, or lifting 267 kg are just a few astonishing examples of world-record performances in Olympic sports. In 2024 the whole world will once again be able to testify ca. 10,500 athletes from 206 National Olympic Committees striving to perform at their very best in 32 different sports. Extraordinary sports performance is multifaceted relying on a well-trained body and a well-trained mind. In this special issue, we aim to uncover...	
</p>
</div>
</div> <div>
<h2><a href="https://www.in-mind.org/blog/post/in-mind-international-has-relaunched">In Mind International has relaunched </a></h2>
<p><a href="https://www.in-mind.org/blog/post/in-mind-international-has-relaunched"></a> </p>
<div>
<h2><a href="https://www.in-mind.org/blog/post/in-mind-international-has-relaunched">In Mind International has relaunched </a></h2>
<p>
	In Mind is a popular-science psychology journal that strives to make psychological knowledge accessible yet offer in-depth texts that relate results to different societal phenomena. The English version has been closed for submissions for a while due to staffing challenges, but still continues to reach many readers. Most find us via Google, and last year alone articles from the English language edition were shown about 8 million times in Google searches and the webpage had about 600 000 separate visits...	
</p>
</div>
</div> <div>
<h2><a href="https://www.in-mind.org/blog/post/in-mind-international-is-relauching-and-we-are-looking-for-people-to-join-our-team">In-Mind International is relauching, and we are looking for people to join our team</a></h2>
<div>
<h2><a href="https://www.in-mind.org/blog/post/in-mind-international-is-relauching-and-we-are-looking-for-people-to-join-our-team">In-Mind International is relauching, and we are looking for people to join our team</a></h2>
<p>
	In-Mind is an online magazine that publishes articles on psychological topics that are of interest to a general audience. I recently took over as editor-in-chief of the international (English) version of the magazine. I am very excited for the task and am eager to begin to publish new content on the site. A lot is happening in the world as well as in the field of psychology, and I see this as a great opportunity to publish texts that are...	
</p>
</div>
</div> <div>
<h2><a href="https://www.in-mind.org/blog/post/why-a-new-popular-press-psychology-book-refrains-from-giving-advice">Why a New Popular Press Psychology Book Refrains from Giving Advice </a></h2>
<div>
<h2><a href="https://www.in-mind.org/blog/post/why-a-new-popular-press-psychology-book-refrains-from-giving-advice">Why a New Popular Press Psychology Book Refrains from Giving Advice </a></h2>
<p>
	During this pandemic winter, many of us will be away from the people we love most. The absence of the physical presence of loved ones deprives us of hugs, physical touch, and feelings of physical and psychological warmth that no amount of Skype or Zoom ever seems to fully replace. In his forthcoming book, Heartwarming , one of the authors of this editorial (Rocha IJzerman) explores the science of why this is, a field called social thermoregulation. This science points...	
</p>
</div>
</div> <div>
<h2><a href="https://www.in-mind.org/blog/post/how-to-bring-the-gezelligheid-this-pandemic-winter-into-your-home-0">How to bring the gezelligheid this pandemic winter into your home</a></h2>
<div>
<h2><a href="https://www.in-mind.org/blog/post/how-to-bring-the-gezelligheid-this-pandemic-winter-into-your-home-0">How to bring the gezelligheid this pandemic winter into your home</a></h2>
<p>
	One of the things we will miss possibly the most this pandemic winter in the Northern Hemisphere is gezelligheid [ɣəˈzɛləxɛit] . No real English equivalent of gezelligheid exists; the closest word in the English vernacular - coziness - still doesn’t capture the same feeling of intimacy and belonging. What does communicate a similar sentiment and is more familiar to US ears is the Danish concept hygge [hʊɡə] and the Swedish concept of lagom . Wikipedia describes gezelligheid as “'conviviality', 'coziness',...	
</p>
</div>
</div> <div>
<h2><a href="https://www.in-mind.org/blog/post/interview-with-academics-for-black-survival-and-wellness">Interview with Academics for Black Survival and Wellness</a></h2>
<div>
<h2><a href="https://www.in-mind.org/blog/post/interview-with-academics-for-black-survival-and-wellness">Interview with Academics for Black Survival and Wellness</a></h2>
<p>
	"Academics cannot stay silent about anti-Black racism, nor can they remain silent in the face of racial violence and injustice. We need to engage in the critical work to dismantle anti-Black racism within ourselves, others, and in the system." - Jeannette Mejia, University of Florida Counseling Psychology Doctoral Student Over the last few weeks, much attention has been directed towards the violence and injustice Black people are facing. In an effort to encourage academics to practice anti-racism and foster welcoming...	
</p>
</div>
</div> <div>
<h2><a href="https://www.in-mind.org/blog/post/hoarding-in-science-no-thanks-openness-and-transparency-in-crisis-mode-and-beyond">Hoarding in science, no thanks. Openness and transparency in crisis mode and beyond</a></h2>
<div>
<h2><a href="https://www.in-mind.org/blog/post/hoarding-in-science-no-thanks-openness-and-transparency-in-crisis-mode-and-beyond">Hoarding in science, no thanks. Openness and transparency in crisis mode and beyond</a></h2>
<p>
	During the current SARS-COV-2 pandemic, the scientific approach is thought to be unable to keep up with the rapid pace at which the crisis is spreading. After developing a research idea, it usually takes years to publish the results. Now, however, things need to move faster. One more reason to embrace open scientific practices. The SARS-COV-2 pandemic poses extreme challenges. How can we help those who have been isolated for weeks? Which vaccine can protect us against SARS-COV-2? What about...	
</p>
</div>
</div>
</div>
<div>
<div>
<h3>blog categories</h3></div>
<div>
<h3>popular blog posts</h3></div>
<div>
<h3>facebook</h3>
</div>
</div>
</div></div>||||I|||| Skip to main content
 select magazine
 * INT
 * DE
 * FR
 * IT
 * NL
 In-Mind channels
 * 
 * 
 * 
 * 
 Menu
 * the magazine
 + Current Issue
 + All Issues
 + Glossary
 + For Authors
 + Magazine Staff
 + Archive
 * blog
 + Blog
 + Blog Staff
 + For Authors
 * book reviews
 + Book Reviews
 + Book Review Staff
 * videos
 + In-Mind Videos
 * the foundation
 + What is In-Mind?
 + Foundation Staff
 + Credits
 * donate
 * contact
 * imprint

 the InMind blog

 hide list read current issue
 Select issue... In-Mind issue 04/2024In-Mind issue 03/2024In-Mind issue 02/2024In-Mind issue 06/2023In-Mind issue 12/2021In-Mind issue 05/2021In-Mind issue 02/2021In-Mind issue 11/2020In-Mind issue 01/2020In-Mind issue 12/2019In-Mind issue 10/2019In-Mind issue 04/2019In-Mind issue 01/2019In-Mind issue 09/2018In-Mind issue 05/2018In-Mind issue 04/2018In-Mind issue 01/2018In-Mind issue 07/2017In-Mind issue 03/2017In-Mind issue 10/2016In-Mind issue 07/2016In-Mind issue 04/2016In-Mind issue 01/2016In-Mind issue 11/2015In-Mind issue 08/2015In-Mind issue 06/2015In-Mind issue 03/2015In-Mind issue 01/2015In-Mind issue 10/2014In-Mind issue 08/2014In-Mind issue 06/2014In-Mind issue 04/2014In-Mind issue 02/2014In-Mind issue 10/2013In-Mind issue 02/2013In-Mind issue 12/2012In-Mind issue 05/2012In-Mind issue 02/2012In-Mind issue 01/2012In-Mind issue 06/2011In-Mind issue 10/2010In-Mind issue 10/2009In-Mind issue 04/2009In-Mind issue 09/2008In-Mind issue 06/2008In-Mind issue 02/2008In-Mind issue 10/2007In-Mind issue 07/2007In-Mind issue 06/2007In-Mind issue 02/2007In-Mind issue 12/2006

 Horoscopes – why we believe in them

 written by Lea Sperlich

 Does nudging have a place in politics? How decision-making contexts inevitably influence our behavior

 written by Mario Herberz

 24.04.2024 | blog categorie(s) - Gender
 written by Silvana Weber

 It’s just a joke, right? Empirical findings on the serious effects of sexist humor

 It’s just a joke, right? Empirical findings on the serious effects of sexist humor

 You can still make a joke, right? Yes, but... misogynistic jokes about women can have negative consequences. What can such humor do to women and men? / more

 22.04.2024 | blog categorie(s) - Other
 written by Christoph Bamberg

 Breakfast is the most important meal of the day…or is it?

 Breakfast is the most important meal of the day…or is it?

 Many people follow the motto “breakfast is the most important meal of the day”. On the other hand, intermittent fasting, which often involves deliberately skipping breakfast, is becoming increasingly popular. How does this fit together? What, how much and when you should eat is a much-discussed topic with countless opinions. For example, who hasn't heard the saying: "Breakfast is the most important meal of the day"? Most people would perhaps attribute this to the wisdom and life experience of their... / more

 04.02.2024 | blog categorie(s) - Happiness & Well-being
 written by Clara Kühner

 How harmful is “always-on” for our well-being? Technology-assisted supplemental work

 How harmful is “always-on” for our well-being? Technology-assisted supplemental work

 Calling a colleague on the way home, finishing a presentation after dinner or checking emails before going to bed - for many employees, this is the norm rather than an exception. But to what extent does being "always on" jeopardize our well-being? And what can help us to benefit from additional work after-hours? For most employees, smartphones, laptops, and co are an integral part of their working lives. In many professions, information and communication technologies such as E-Mail and messenger... / more

 16.11.2023 | blog categorie(s) - Announcements and News
 written by Lisa Musculus

 Olympic Mind: An International Special Issue on The Psychology of Extraordinary Sports Performance

 Olympic Mind: An International Special Issue on The Psychology of Extraordinary Sports Performance

 Performing the Biles 3 in gymnastics, running 100m in 9.58 seconds, or lifting 267 kg are just a few astonishing examples of world-record performances in Olympic sports. In 2024 the whole world will once again be able to testify ca. 10,500 athletes from 206 National Olympic Committees striving to perform at their very best in 32 different sports. Extraordinary sports performance is multifaceted relying on a well-trained body and a well-trained mind. In this special issue, we aim to uncover... / more

 16.11.2023 | blog categorie(s) - Announcements and News
 written by Sofia Calderon, Jana Dreston, Stella Wernicke, Maren Flottmann, Kaitlyn Werner & Malin Ekelund

 In Mind International has relaunched

 In Mind International has relaunched

 In Mind is a popular-science psychology journal that strives to make psychological knowledge accessible yet offer in-depth texts that relate results to different societal phenomena. The English version has been closed for submissions for a while due to staffing challenges, but still continues to reach many readers. Most find us via Google, and last year alone articles from the English language edition were shown about 8 million times in Google searches and the webpage had about 600 000 separate visits... / more

 28.06.2023 | blog categorie(s) - Announcements and News
 written by Sofia Calderon

 In-Mind International is relauching, and we are looking for people to join our team

 In-Mind International is relauching, and we are looking for people to join our team

 In-Mind is an online magazine that publishes articles on psychological topics that are of interest to a general audience. I recently took over as editor-in-chief of the international (English) version of the magazine. I am very excited for the task and am eager to begin to publish new content on the site. A lot is happening in the world as well as in the field of psychology, and I see this as a great opportunity to publish texts that are... / more

 09.03.2021 | blog categorie(s) - Generalist Grab Bag
 written by Patrick Forscher & Hans IJzerman

 Why a New Popular Press Psychology Book Refrains from Giving Advice

 keywords:
 Popular Science, Thermregulation, science, Books

 Why a New Popular Press Psychology Book Refrains from Giving Advice

 During this pandemic winter, many of us will be away from the people we love most. The absence of the physical presence of loved ones deprives us of hugs, physical touch, and feelings of physical and psychological warmth that no amount of Skype or Zoom ever seems to fully replace. In his forthcoming book, Heartwarming , one of the authors of this editorial (Rocha IJzerman) explores the science of why this is, a field called social thermoregulation. This science points... / more

 17.02.2021 | blog categorie(s) - Generalist Grab Bag
 written by Hans IJzerman & Olivier Dujols

 How to bring the gezelligheid this pandemic winter into your home

 keywords:
 relationships, warmth, closeness, gezelligheid, Heartwarming, Hans Rocha IJzerman

 How to bring the gezelligheid this pandemic winter into your home

 One of the things we will miss possibly the most this pandemic winter in the Northern Hemisphere is gezelligheid [ɣəˈzɛləxɛit] . No real English equivalent of gezelligheid exists; the closest word in the English vernacular - coziness - still doesn’t capture the same feeling of intimacy and belonging. What does communicate a similar sentiment and is more familiar to US ears is the Danish concept hygge [hʊɡə] and the Swedish concept of lagom . Wikipedia describes gezelligheid as “'conviviality', 'coziness',... / more

 18.06.2020 | blog categorie(s) - Big Questions in Society, Current Events, Race & Ethnicity
 written by Regine Louis & Scott Stroud

 Interview with Academics for Black Survival and Wellness

 keywords:
 race, racism, black lives matter, wellness

 Interview with Academics for Black Survival and Wellness

 "Academics cannot stay silent about anti-Black racism, nor can they remain silent in the face of racial violence and injustice. We need to engage in the critical work to dismantle anti-Black racism within ourselves, others, and in the system." - Jeannette Mejia, University of Florida Counseling Psychology Doctoral Student Over the last few weeks, much attention has been directed towards the violence and injustice Black people are facing. In an effort to encourage academics to practice anti-racism and foster welcoming... / more

 10.06.2020 | blog categorie(s) - Big Questions in Society
 written by Rima-Maria Rahal & Tobias Heycke

 Hoarding in science, no thanks. Openness and transparency in crisis mode and beyond

 keywords:
 covid-19, open science, scientific practice, crisis response

 Hoarding in science, no thanks. Openness and transparency in crisis mode and beyond

 During the current SARS-COV-2 pandemic, the scientific approach is thought to be unable to keep up with the rapid pace at which the crisis is spreading. After developing a research idea, it usually takes years to publish the results. Now, however, things need to move faster. One more reason to embrace open scientific practices. The SARS-COV-2 pandemic poses extreme challenges. How can we help those who have been isolated for weeks? Which vaccine can protect us against SARS-COV-2? What about... / more

 page 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | | next ›

 blog categories

 * Announcements and News (9)
 * Big Questions in Society (13)
 * Culture (13)
 * Current Events (9)
 * Demystifying Mental Health (3)
 * Gender (6)
 * Generalist Grab Bag (7)
 * Happiness & Well-being (11)
 * Interdisciplinary Inquiries (2)
 * Meaning Making (7)
 * Other (2)
 * Political Psychology (11)
 * Popular Culture Series (2)
 * Race & Ethnicity (7)
 * Romance & Friendship (3)
 * Self-Control & Decision-Making (3)
 * Social Influence & Negotiations (3)
 * Solid Science (8)
 * Violence & Aggression (7)

 popular blog posts

 Does parental disapproval lead to love or dissolution? The Romeo and Juliet effect vs. the social network effect

 Look at me! (Or don’t): Of society and showing off on Facebook

 Re-thinking how we think about conformity

 facebook

 the magazine

 * Current issue
 * All issues
 * Glossary
 * For authors
 * Magazine staff

 other

 * Glossary
 * Book reviews
 * In-Mind blog

 the foundation

 * What is In-Mind
 * Foundation staff
 * Credits

 misc

 * Contact
 * Imprint
 * Newsletter
24.04.2024 | 
	blog categorie(s) - Gender 
 It’s just a joke, right? Empirical findings on the serious effects of sexist humor 
   
 It’s just a joke, right? Empirical findings on the serious effects of sexist humor 
	You can still make a joke, right? Yes, but... misogynistic jokes about women can have negative consequences. What can such humor do to women and men?	
 
	22.04.2024 | 
	blog categorie(s) - Other 
 Breakfast is the most important meal of the day…or is it? 
   
 Breakfast is the most important meal of the day…or is it? 
	Many people follow the motto “breakfast is the most important meal of the day”. On the other hand, intermittent fasting, which often involves deliberately skipping breakfast, is becoming increasingly popular. How does this fit together? What, how much and when you should eat is a much-discussed topic with countless opinions. For example, who hasn't heard the saying: "Breakfast is the most important meal of the day"? Most people would perhaps attribute this to the wisdom and life experience of their...	
 
 How harmful is “always-on” for our well-being? Technology-assisted supplemental work 
   
 How harmful is “always-on” for our well-being? Technology-assisted supplemental work 
	Calling a colleague on the way home, finishing a presentation after dinner or checking emails before going to bed - for many employees, this is the norm rather than an exception. But to what extent does being "always on" jeopardize our well-being? And what can help us to benefit from additional work after-hours? For most employees, smartphones, laptops, and co are an integral part of their working lives. In many professions, information and communication technologies such as E-Mail and messenger...	
 
 Olympic Mind: An International Special Issue on The Psychology of Extraordinary Sports Performance 
   
 Olympic Mind: An International Special Issue on The Psychology of Extraordinary Sports Performance 
	Performing the Biles 3 in gymnastics, running 100m in 9.58 seconds, or lifting 267 kg are just a few astonishing examples of world-record performances in Olympic sports. In 2024 the whole world will once again be able to testify ca. 10,500 athletes from 206 National Olympic Committees striving to perform at their very best in 32 different sports. Extraordinary sports performance is multifaceted relying on a well-trained body and a well-trained mind. In this special issue, we aim to uncover...	
 
 In Mind International has relaunched  
   
 In Mind International has relaunched  
	In Mind is a popular-science psychology journal that strives to make psychological knowledge accessible yet offer in-depth texts that relate results to different societal phenomena. The English version has been closed for submissions for a while due to staffing challenges, but still continues to reach many readers. Most find us via Google, and last year alone articles from the English language edition were shown about 8 million times in Google searches and the webpage had about 600 000 separate visits...	
 
 In-Mind International is relauching, and we are looking for people to join our team 
 In-Mind International is relauching, and we are looking for people to join our team 
	In-Mind is an online magazine that publishes articles on psychological topics that are of interest to a general audience. I recently took over as editor-in-chief of the international (English) version of the magazine. I am very excited for the task and am eager to begin to publish new content on the site. A lot is happening in the world as well as in the field of psychology, and I see this as a great opportunity to publish texts that are...	
 
 Why a New Popular Press Psychology Book Refrains from Giving Advice  
 Why a New Popular Press Psychology Book Refrains from Giving Advice  
	During this pandemic winter, many of us will be away from the people we love most. The absence of the physical presence of loved ones deprives us of hugs, physical touch, and feelings of physical and psychological warmth that no amount of Skype or Zoom ever seems to fully replace. In his forthcoming book, Heartwarming , one of the authors of this editorial (Rocha IJzerman) explores the science of why this is, a field called social thermoregulation. This science points...	
 
 How to bring the gezelligheid this pandemic winter into your home 
 How to bring the gezelligheid this pandemic winter into your home 
	One of the things we will miss possibly the most this pandemic winter in the Northern Hemisphere is gezelligheid [ɣəˈzɛləxɛit] . No real English equivalent of gezelligheid exists; the closest word in the English vernacular - coziness - still doesn’t capture the same feeling of intimacy and belonging. What does communicate a similar sentiment and is more familiar to US ears is the Danish concept hygge [hʊɡə] and the Swedish concept of lagom . Wikipedia describes gezelligheid as “'conviviality', 'coziness',...	
 
 Interview with Academics for Black Survival and Wellness 
 Interview with Academics for Black Survival and Wellness 
	"Academics cannot stay silent about anti-Black racism, nor can they remain silent in the face of racial violence and injustice. We need to engage in the critical work to dismantle anti-Black racism within ourselves, others, and in the system." - Jeannette Mejia, University of Florida Counseling Psychology Doctoral Student Over the last few weeks, much attention has been directed towards the violence and injustice Black people are facing. In an effort to encourage academics to practice anti-racism and foster welcoming...	
 
 Hoarding in science, no thanks. Openness and transparency in crisis mode and beyond 
 Hoarding in science, no thanks. Openness and transparency in crisis mode and beyond 
	During the current SARS-COV-2 pandemic, the scientific approach is thought to be unable to keep up with the rapid pace at which the crisis is spreading. After developing a research idea, it usually takes years to publish the results. Now, however, things need to move faster. One more reason to embrace open scientific practices. The SARS-COV-2 pandemic poses extreme challenges. How can we help those who have been isolated for weeks? Which vaccine can protect us against SARS-COV-2? What about...	
blog categories 
popular blog posts 
facebook
the InMind blog | In-Mind
https://www.in-mind.org/blog
https://www.in-mind.org/blog/post/why-a-new-popular-press-psychology-book-refrains-from-giving-advice
the InMind blog
2021-03-09
Patrick Forscher Patrick S Forscher is a research scientist; PC
<div><div>
	<p>During this pandemic winter, many of us will be away from the people we love most. The absence of the physical presence of loved ones deprives us of hugs, physical touch, and feelings of physical and psychological 
<a href="/glossary/letter_w#warmth">warmth</a> that no amount of Skype or Zoom ever seems to fully replace.</p>
<p>In his forthcoming book, <em>Heartwarming</em>, one of the authors of this editorial (Rocha IJzerman) explores the science of why this is, a 
<a href="/glossary/letter_f#field">field</a> called social thermoregulation. This science points to promising technologies like the <a href="https://embrlabs.com/products/embr-wave">Embr Wave</a>, smartphone-controlled 
<a href="/glossary/letter_w#warmth">warmth</a>-producing bracelet, that seem to provide ways to compensate for the physical and psychological 
<a href="/glossary/letter_w#warmth">warmth</a> that Skype and Zoom lack. Yet the book stops short of recommending these technologies, or indeed offering any concrete advice at all. Why is this?</p>
<p><a href="https://in-mind.org/sites/default/files/background.jpg"></a></p>
<p>One reason stems from the sordid tale of Brian Wansink. A former marketing researcher at Cornell, Wansink excelled at conducting clever-sounding studies on the psychology of eating and packaging them into bit-size, actionable pieces of advice tailor-made for public consumption. He would then pitch this advice in outlets ranging from <em><a href="https://www.oprah.com/health/lose-weight-with-a-diet-makeover/">O Magazine</a></em> to the <em><a href="https://www.today.com/health/kitchen-makeover-8-small-changes-help-you-lose-weight-t114944">Today Show</a></em>. His work even served as the basis for federal policy in the form of the <a href="https://snaped.fns.usda.gov/library/materials/smarter-lunchrooms-movement">Smarter Lunchroom Movement</a>. Yet, proper scrutiny revealed a <a href="https://www.buzzfeednews.com/article/stephaniemlee/brian-wansink-cornell-p-hacking">shocking level of sloppiness</a> in Wansink’s research. Although Wansink had spun his research into advice for millions of people and a federal program funded to the tune of <a href="https://www.buzzfeednews.com/article/stephaniemlee/brian-wansink-cornell-smarter-lunchrooms-flawed-data#.mxEgAELEv">$22 million dollars</a>, all these products were built on foundations of sand.</p>
<p>Nor is Wansink’s work the only example. Take the example of research on implicit bias. Defined as a set of fast, relatively automatic mental pairings between 
<a href="/glossary/letter_s#social_group">social groups</a> and other concepts, the concept of implicit bias was popularized largely due to the efforts of Anthony Greenwald and Mahzarin Banaji. These two researchers developed a test, the Implicit Association Test, and demonstrated it in a <a href="https://www.washington.edu/news/1998/09/29/roots-of-unconscious-prejudice-affect-90-to-95-percent-of-people-psychologists-demonstrate-at-press-conference/">press conference</a> in 1998 in which they claimed to have data demonstrating that 90 to 95 percent of people harbor “the unconscious roots of 
<a href="/glossary/letter_p#prejudice">prejudice</a>. The test became an immediate 
<a href="/glossary/letter_s#sensation">sensation</a>, receiving favorable coverage by the wildly popular science journalist <a href="https://www.thecut.com/2017/01/psychologys-racism-measuring-tool-isnt-up-to-the-job.html">Malcolm Gladwell</a>, NPR correspondent <a href="https://www.thecut.com/2017/01/psychologys-racism-measuring-tool-isnt-up-to-the-job.html">Shankar Vedantam</a>, and, more recently, even then-presidential candidate Hillary Clinton in the rarified halls of a <a href="https://www.washingtonpost.com/video/politics/clinton-on-implicit-bias-in-policing/2016/09/26/46e1e88c-8441-11e6-b57d-dd49277af02f_video.html">US presidential debate</a>.</p>
<p>Yet a closer look at the evidence on implicit bias reveals a 
<a href="/glossary/letter_f#field">field</a> that is woefully unready for application. The Implicit Association Test is plagued with measurement problems (as are, for that matter, <a href="https://psyarxiv.com/d5zn8">other</a> <a href="https://psyarxiv.com/ge3k7">alternative</a> measures of implicit bias). For example, too often the same person who takes the test multiple times gets a different result, probably because the test is <a href="https://journals.sagepub.com/doi/abs/10.1177/1745691620931492">contaminated</a> by large amounts of measurement error. The relationship between people’s score on the IAT and their actual behavior is also <a href="https://journals.sagepub.com/doi/abs/10.1177/1745691619863798">quite weak</a>. Finally, a <a href="https://psyarxiv.com/dv8tu/">closer look</a> at studies that ought to be most relevant to policy -- those that try to change implicit bias -- reveals that this 
<a href="/glossary/letter_f#field">field</a> relies very heavily on student samples, rarely assesses whether the changes persist, and contains little to no evidence that changes in implicit bias lead to changes in behavior.</p>
<p>In <em>Heartwarming</em>, Rocha IJzerman wanted to avoid the mistakes of Brian Wansink, Tony Greenwald, and Mahzarin Banaji. We believe psychology can learn from other research fields, like rocket science and drug development, that excel at translating research findings into safe and effective applications. Rocket science uses a framework called the <a href="https://www.nasa.gov/directorates/heo/scan/engineering/technology/txt_accordion1.html">Technology Readiness Levels</a> to evaluate the state of the evidence behind an application and to guide the research program. Drug development uses a series of <a href="https://www.fda.gov/patients/drug-development-process/step-3-clinical-research#:~:text=Clinical%20trials%20follow%20a%20typical,large%20scale%2C%20Phase%203%20studies.">phased trials</a> to evaluate both the effectiveness and the 
<a href="/glossary/letter_s#safety">safety</a> of a novel drug. As we have seen from the accomplishments of these fields -- from putting people on the moon to creating a safe and effective Covid-19 vaccine in record-breaking time -- this systematic process <em>works</em>.</p>
<p><em>Heartwarming</em> judges the science of social thermoregulation using standards closer to those of rocket science and drug development rather than those of Brian Wansink. According to those standards, social thermoregulation, though promising, is not yet ready to serve as the basis of advice and other applications. Our <a href="https://www.dropbox.com/s/07yc84jkcb1qlgw/IJzerman%20et%20al%20May%203%20Meta-Analysis%20submission%20psyarxiv.pdf?dl=0">unpublished quantitative review</a> suggests the available evidence on social thermoregulation can support the broad conclusion of <em>Heartwarming</em>: there are intriguing links between the regulation of temperature and social relationships that promise to provide unique insights into the roots of human sociality. Yet social thermoregulation research still largely relies on samples of participants that are very peculiar -- college students from Europe and the United States. We also know little about the effectiveness of different dosages of 
<a href="/glossary/letter_w#warmth">warmth</a> for changing psychology. Nor, as would be routine in drug development, have the technologies based on social thermoregulation research been evaluated in household settings for things like unintended side effects.</p>
<p>Thus, while <em>Heartwarming</em> does present science that is promising enough that it <em>could in the future</em> serve as the basis for advice and application, it does so with caveats. The book therefore notes where social thermoregulation research relies too heavily on unusual and non-representative samples, and it also notes where its findings have not been subjected to the type of big-team, large-scale testing that you might get in drug development.</p>
<p>Psychology may yet initiate the reforms necessary to make it robust enough for application. Psychology researchers (<a href="https://www.nature.com/articles/s41562-020-00990-w">ourselves included</a>) have developed our own frameworks to help evaluate just when psychology evidence is ready for application. Moreover, psychology researchers have <a href="https://psysciacc.org/">created organizations</a> and <a href="https://psyarxiv.com/2mdxh">proposed broad-based reforms</a> to make the big, team-style research more commonplace.</p>
<p>In the meantime, beware psychologists bearing overconfident advice. You are bound to be disappointed.</p>
<p><a href="https://corelab.io/">Hans Rocha IJzerman</a> is the author of <em><a href="https://wwnorton.com/books/heartwarming">Heartwarming: How Our Inner Thermostat Made Us Human</a></em>.</p>
 	</div>||||I|||| Skip to main content
 select magazine
 * INT
 * DE
 * FR
 * IT
 * NL
 In-Mind channels
 * 
 * 
 * 
 * 
 Menu
 * the magazine
 + Current Issue
 + All Issues
 + Glossary
 + For Authors
 + Magazine Staff
 + Archive
 * blog
 + Blog
 + Blog Staff
 + For Authors
 * book reviews
 + Book Reviews
 + Book Review Staff
 * videos
 + In-Mind Videos
 * the foundation
 + What is In-Mind?
 + Foundation Staff
 + Credits
 * donate
 * contact
 * imprint

 the InMind blog

 hide list read current issue
 Select issue... In-Mind issue 12/2021In-Mind issue 05/2021In-Mind issue 2/2021In-Mind issue 11/2020In-Mind issue 01/2020In-Mind issue 12/2019In-Mind issue 10/2019In-Mind issue 04/2019In-Mind issue 01/2019In-Mind issue 09/2018In-Mind issue 05/2018In-Mind issue 04/2018In-Mind issue 01/2018In-Mind issue 07/2017In-Mind issue 03/2017In-Mind issue 10/2016In-Mind issue 07/2016In-Mind issue 04/2016In-Mind issue 01/2016In-Mind issue 11/2015In-Mind issue 08/2015In-Mind issue 06/2015In-Mind issue 03/2015In-Mind issue 01/2015In-Mind issue 10/2014In-Mind issue 08/2014In-Mind issue 06/2014In-Mind issue 04/2014In-Mind issue 02/2014In-Mind issue 10/2013In-Mind issue 02/2013In-Mind issue 12/2012In-Mind issue 05/2012In-Mind issue 02/2012In-Mind issue 01/2012In-Mind issue 06/2011In-Mind issue 10/2010In-Mind issue 10/2009In-Mind issue 04/2009In-Mind issue 09/2008In-Mind issue 06/2008In-Mind issue 02/2008In-Mind issue 10/2007In-Mind issue 07/2007In-Mind issue 06/2007In-Mind issue 02/2007In-Mind issue 12/2006

 Are Victim or Eyewitness Statements Credible? Several Ways to Check Them

 written by Nurul Arbiyah, Henry Otgaar & Eric Rassin

 09.03.2021 | Generalist Grab Bag
 Tweet

 Why a New Popular Press Psychology Book Refrains from Giving Advice

 * written by Patrick Forscher & Hans IJzerman

 During this pandemic winter, many of us will be away from the people we love most. The absence of the physical presence of loved ones deprives us of hugs, physical touch, and feelings of physical and psychological warmth that no amount of Skype or Zoom ever seems to fully replace.

 In his forthcoming book, Heartwarming, one of the authors of this editorial (Rocha IJzerman) explores the science of why this is, a field called social thermoregulation. This science points to promising technologies like the Embr Wave, smartphone-controlled warmth-producing bracelet, that seem to provide ways to compensate for the physical and psychological warmth that Skype and Zoom lack. Yet the book stops short of recommending these technologies, or indeed offering any concrete advice at all. Why is this?

 One reason stems from the sordid tale of Brian Wansink. A former marketing researcher at Cornell, Wansink excelled at conducting clever-sounding studies on the psychology of eating and packaging them into bit-size, actionable pieces of advice tailor-made for public consumption. He would then pitch this advice in outlets ranging from O Magazine to the Today Show. His work even served as the basis for federal policy in the form of the Smarter Lunchroom Movement. Yet, proper scrutiny revealed a shocking level of sloppiness in Wansink’s research. Although Wansink had spun his research into advice for millions of people and a federal program funded to the tune of $22 million dollars, all these products were built on foundations of sand.

 Nor is Wansink’s work the only example. Take the example of research on implicit bias. Defined as a set of fast, relatively automatic mental pairings between social groups and other concepts, the concept of implicit bias was popularized largely due to the efforts of Anthony Greenwald and Mahzarin Banaji. These two researchers developed a test, the Implicit Association Test, and demonstrated it in a press conference in 1998 in which they claimed to have data demonstrating that 90 to 95 percent of people harbor “the unconscious roots of prejudice. The test became an immediate sensation, receiving favorable coverage by the wildly popular science journalist Malcolm Gladwell, NPR correspondent Shankar Vedantam, and, more recently, even then-presidential candidate Hillary Clinton in the rarified halls of a US presidential debate.

 Yet a closer look at the evidence on implicit bias reveals a field that is woefully unready for application. The Implicit Association Test is plagued with measurement problems (as are, for that matter, other alternative measures of implicit bias). For example, too often the same person who takes the test multiple times gets a different result, probably because the test is contaminated by large amounts of measurement error. The relationship between people’s score on the IAT and their actual behavior is also quite weak. Finally, a closer look at studies that ought to be most relevant to policy -- those that try to change implicit bias -- reveals that this field relies very heavily on student samples, rarely assesses whether the changes persist, and contains little to no evidence that changes in implicit bias lead to changes in behavior.

 In Heartwarming, Rocha IJzerman wanted to avoid the mistakes of Brian Wansink, Tony Greenwald, and Mahzarin Banaji. We believe psychology can learn from other research fields, like rocket science and drug development, that excel at translating research findings into safe and effective applications. Rocket science uses a framework called the Technology Readiness Levels to evaluate the state of the evidence behind an application and to guide the research program. Drug development uses a series of phased trials to evaluate both the effectiveness and the safety of a novel drug. As we have seen from the accomplishments of these fields -- from putting people on the moon to creating a safe and effective Covid-19 vaccine in record-breaking time -- this systematic process works.

 Heartwarming judges the science of social thermoregulation using standards closer to those of rocket science and drug development rather than those of Brian Wansink. According to those standards, social thermoregulation, though promising, is not yet ready to serve as the basis of advice and other applications. Our unpublished quantitative review suggests the available evidence on social thermoregulation can support the broad conclusion of Heartwarming: there are intriguing links between the regulation of temperature and social relationships that promise to provide unique insights into the roots of human sociality. Yet social thermoregulation research still largely relies on samples of participants that are very peculiar -- college students from Europe and the United States. We also know little about the effectiveness of different dosages of warmth for changing psychology. Nor, as would be routine in drug development, have the technologies based on social thermoregulation research been evaluated in household settings for things like unintended side effects.

 Thus, while Heartwarming does present science that is promising enough that it could in the future serve as the basis for advice and application, it does so with caveats. The book therefore notes where social thermoregulation research relies too heavily on unusual and non-representative samples, and it also notes where its findings have not been subjected to the type of big-team, large-scale testing that you might get in drug development.

 Psychology may yet initiate the reforms necessary to make it robust enough for application. Psychology researchers (ourselves included) have developed our own frameworks to help evaluate just when psychology evidence is ready for application. Moreover, psychology researchers have created organizations and proposed broad-based reforms to make the big, team-style research more commonplace.

 In the meantime, beware psychologists bearing overconfident advice. You are bound to be disappointed.

 Hans Rocha IJzerman is the author of Heartwarming: How Our Inner Thermostat Made Us Human.

 hide user comments show user comments

 article author(s)

 Patrick Forscher

 Patrick S. Forscher is a research scientist at the LIP/PC2S in the Université ...more

 Hans IJzerman

 Hans (Rocha) IJzerman is an Associate Professor of Psychology at Université Grenoble Alpes and Institut Universitaire de France. He is the author of...more

 article keywords

 * Books
 * Popular Science
 * science
 * Thermregulation

 blog categories

 * Announcements and News (6)
 * Big Questions in Society (13)
 * Culture (13)
 * Current Events (9)
 * Demystifying Mental Health (3)
 * Gender (5)
 * Generalist Grab Bag (7)
 * Happiness & Well-being (10)
 * Interdisciplinary Inquiries (2)
 * Meaning Making (7)
 * Other (1)
 * Political Psychology (11)
 * Popular Culture Series (2)
 * Race & Ethnicity (7)
 * Romance & Friendship (3)
 * Self-Control & Decision-Making (3)
 * Social Influence & Negotiations (3)
 * Solid Science (8)
 * Violence & Aggression (7)

 popular blog posts

 Does parental disapproval lead to love or dissolution? The Romeo and Juliet effect vs. the social network effect

 Look at me! (Or don’t): Of society and showing off on Facebook

 Re-thinking how we think about conformity

 keyword cloud

 Aggression culture embodiment emotion gender morality motivation prejudice romantic relationships social identity stereotypes well-being
 More

 facebook

 the magazine

 * Current issue
 * All issues
 * Glossary
 * For authors
 * Magazine staff

 other

 * Glossary
 * Book reviews
 * In-Mind blog

 the foundation

 * What is In-Mind
 * Foundation staff
 * Credits

 misc

 * Contact
 * Imprint
 * Newsletter
During this pandemic winter, many of us will be away from the people we love most. The absence of the physical presence of loved ones deprives us of hugs, physical touch, and feelings of physical and psychological 
warmth that no amount of Skype or Zoom ever seems to fully replace.
In his forthcoming book, Heartwarming, one of the authors of this editorial (Rocha IJzerman) explores the science of why this is, a 
field called social thermoregulation. This science points to promising technologies like the Embr Wave, smartphone-controlled 
warmth-producing bracelet, that seem to provide ways to compensate for the physical and psychological 
warmth that Skype and Zoom lack. Yet the book stops short of recommending these technologies, or indeed offering any concrete advice at all. Why is this?
   
One reason stems from the sordid tale of Brian Wansink. A former marketing researcher at Cornell, Wansink excelled at conducting clever-sounding studies on the psychology of eating and packaging them into bit-size, actionable pieces of advice tailor-made for public consumption. He would then pitch this advice in outlets ranging from  O Magazine  to the  Today Show . His work even served as the basis for federal policy in the form of the Smarter Lunchroom Movement. Yet, proper scrutiny revealed a shocking level of sloppiness in Wansink’s research. Although Wansink had spun his research into advice for millions of people and a federal program funded to the tune of $22 million dollars, all these products were built on foundations of sand.
Nor is Wansink’s work the only example. Take the example of research on implicit bias. Defined as a set of fast, relatively automatic mental pairings between 
social groups and other concepts, the concept of implicit bias was popularized largely due to the efforts of Anthony Greenwald and Mahzarin Banaji. These two researchers developed a test, the Implicit Association Test, and demonstrated it in a press conference in 1998 in which they claimed to have data demonstrating that 90 to 95 percent of people harbor “the unconscious roots of 
prejudice. The test became an immediate 
sensation, receiving favorable coverage by the wildly popular science journalist Malcolm Gladwell, NPR correspondent Shankar Vedantam, and, more recently, even then-presidential candidate Hillary Clinton in the rarified halls of a US presidential debate.
Yet a closer look at the evidence on implicit bias reveals a 
field that is woefully unready for application. The Implicit Association Test is plagued with measurement problems (as are, for that matter, other alternative measures of implicit bias). For example, too often the same person who takes the test multiple times gets a different result, probably because the test is contaminated by large amounts of measurement error. The relationship between people’s score on the IAT and their actual behavior is also quite weak. Finally, a closer look at studies that ought to be most relevant to policy -- those that try to change implicit bias -- reveals that this 
field relies very heavily on student samples, rarely assesses whether the changes persist, and contains little to no evidence that changes in implicit bias lead to changes in behavior.
In Heartwarming, Rocha IJzerman wanted to avoid the mistakes of Brian Wansink, Tony Greenwald, and Mahzarin Banaji. We believe psychology can learn from other research fields, like rocket science and drug development, that excel at translating research findings into safe and effective applications. Rocket science uses a framework called the Technology Readiness Levels to evaluate the state of the evidence behind an application and to guide the research program. Drug development uses a series of phased trials to evaluate both the effectiveness and the 
safety of a novel drug. As we have seen from the accomplishments of these fields -- from putting people on the moon to creating a safe and effective Covid-19 vaccine in record-breaking time -- this systematic process works.
 Heartwarming judges the science of social thermoregulation using standards closer to those of rocket science and drug development rather than those of Brian Wansink. According to those standards, social thermoregulation, though promising, is not yet ready to serve as the basis of advice and other applications. Our unpublished quantitative review suggests the available evidence on social thermoregulation can support the broad conclusion of Heartwarming: there are intriguing links between the regulation of temperature and social relationships that promise to provide unique insights into the roots of human sociality. Yet social thermoregulation research still largely relies on samples of participants that are very peculiar -- college students from Europe and the United States. We also know little about the effectiveness of different dosages of 
warmth for changing psychology. Nor, as would be routine in drug development, have the technologies based on social thermoregulation research been evaluated in household settings for things like unintended side effects.
Thus, while Heartwarming does present science that is promising enough that it could in the future serve as the basis for advice and application, it does so with caveats. The book therefore notes where social thermoregulation research relies too heavily on unusual and non-representative samples, and it also notes where its findings have not been subjected to the type of big-team, large-scale testing that you might get in drug development.
Psychology may yet initiate the reforms necessary to make it robust enough for application. Psychology researchers (ourselves included) have developed our own frameworks to help evaluate just when psychology evidence is ready for application. Moreover, psychology researchers have created organizations and proposed broad-based reforms to make the big, team-style research more commonplace.
In the meantime, beware psychologists bearing overconfident advice. You are bound to be disappointed.
 Hans Rocha IJzerman is the author of  Heartwarming: How Our Inner Thermostat Made Us Human .
the InMind blog
https://www.in-mind.org/blog/post/why-a-new-popular-press-psychology-book-refrains-from-giving-advice