Feed aggregator

1. As We May Code — why aren’t we — ostensibly the people writing software — doing more with AI in our day-to-day? Why are things like TabNine and Kite so often seen as curiosities instead of game-changers? If you take seriously the idea that ai will fundamentally change the nature of many occupations in the coming decade, what reason do you have to believe that you’ll be immune from that because you work in software? Looking at the code you’ve been paid to write over the past few years, how much of that can you honestly say is truly novel? Regular readers know this is consonant with my beliefs, though I’m not as bullish on RDF and the SemWeb as Matt may be. One of his closing lines also resonated: The truth is, I’ve come around to thinking that programming isn’t the most important thing for programmers to pay attention to right now.
2. DataScript as a Lingua Franca for Domain Modeling — represent our Domain Model declaratively, as an in-program data structure (a ‘meta-database’); derive the ‘machine’ behaviour generically from this representation. Implemented using DataScript, an in-memory database / data-structure, available as a library on the JVM or JavaScript, which takes inspiration from the Datomic database.
3. Structured Evidential Argumentation System — We have developed a new methodology that retains the ease-of-use, familiarity, and (some of) the free-form nature of informal methods, while benefiting from the rigor, structure, and potential for automation characteristic of formal methods. Our approach aims to foster thoughtful and timely analysis through the introduction of structure, and collaboration through access to the corporate memory of current and past analytic results. Unchanged since 2007, according to this website. I wish it were open sourced to experiment with and build on.
4. Debugging Incidents in Google’s Distributed Systems — This article covers the outcomes of research performed in 2019 on how engineers at Google debug production issues, including the types of tools, high-level strategies, and low-level tasks that engineers use in varying combinations to debug effectively. It examines the research approach used to capture data, summarizing the common engineering journeys for production investigations and sharing examples of how experts debug complex distributed systems. Finally, the article extends the Google specifics of this research to provide some practical strategies that you can apply in your organization.

1. wiki.js — Portable open-source Javascript featureful wiki.
2. Neural Embedding + Locality Sensitive Hashing for Logs — Our diff solution involves embedding each line into a low dimensional vector and (optionally “fine-tuning” or updating the embedding model at the same time), assigning it to a cluster, and identifying lines in different clusters as “different”. Locality sensitive hashing is a probabilistic algorithm that permits constant time cluster assignment and near-constant time nearest neighbors search. A clever way of getting approximate search in logs.
3. Remove Chaos — Good leaders can walk into a situation where people have lost track of their goals and get everyone aligned on a clear path forward. They remove unimportant details, distill complex situations to their essence, and get the right decision-maker to make a call – even if it’s not them. They’re able to not only stop bad plans before it’s too late, but get them moving again in the right direction. It’s a life-long quest.
4. 1998 Zoom — An hilarious infomercial for a 1998 version of Zoom.

Largely on the impetus of the Black Lives Matter movement, the public’s response to the murder of George Floyd, and the subsequent demonstrations, we’ve seen increased concern about the use of facial identification in policing.

First, in a highly publicized wave of announcements, IBM, Microsoft, and Amazon have announced that they will not sell face recognition technology to police forces. IBM’s announcement went the furthest; they’re withdrawing from face recognition research and product development. Amazon’s statement was much more limited; they’re putting a one-year moratorium on the police use of their Rekognition product, and hoping that Congress will pass regulation on the use of face recognition in the meantime.

These statements are fine, as far as they go. As many point out, Amazon and Microsoft are just passing the buck to Congress, which isn’t likely to do anything substantial. (As far as I know, Amazon is still partnering with local police forces on their Ring smart lock, which includes a camera.) And, as others have pointed out, IBM, Microsoft, and Amazon are not the most important companies that supply face recognition technology to law enforcement. That’s dominated by a number of less prominent companies, of which the most visible are Palantir and Clearview AI. I suspect the executives at those companies are smiling; perhaps IBM, Microsoft, and Amazon aren’t the most important players, but their departure (even if only temporary) means that there’s less competition.

So, much as I approve companies pulling back from products that are used unethically, we also have to be clear about what this actually accomplishes: not much. Other companies that are less concerned about ethics will fill the gap.

Another response is increased efforts within cities to ban the use of face recognition technologies by police. That trend, of course, isn’t new; San Francisco, Oakland, Boston, and a number of other cities have instituted such bans. Accuracy is an issue—not just for people of color, but for anyone. London’s police chief is on record as saying that he’s “completely comfortable” with the use of face recognition technology, despite their department’s 98% false positive rate. I’ve seen similar statements, and similar false positive rates, from other departments.

We’ve also seen the first known case of a person falsely arrested because of face recognition. “First known case” is extremely important in this context; the victim only found out that he was targeted by face recognition because he overheard a conversation between police officers. We need to ask: how many people have already been arrested, imprisoned, and even convicted on the basis of incorrect face recognition? I am sure that number isn’t zero, and I suspect it is shockingly large.

City-wide bans on the use of face recognition by police are a step in the right direction; statewide and national legislation would be better; but I think we have to ask the harder question. Given that police response to the protests over George Floyd’s murder has revealed that, in many cities, law enforcement is essentially lawless, will these regulations have any effect? Or will they just be ignored? My guess is “ignored.”

That brings me to my point: given that companies backing off from sales of face recognition products, and local regulation of the use of these products, are praiseworthy but unlikely to be effective, what other response is possible? How do we shift the balance of power between surveillors and surveillees? What can be done to subvert these systems?

There are two kinds of responses. First, the use of extreme fashion. CVDazzle is one site that shows how fashion can be used to defeat face detection. There are others, such as Juggalo makeup. If you don’t like these rather extreme looks, remember that researchers have shown that even a few altered pixels can defeat image recognition, changing a stop sign into something else. Can a simple “birthmark,” applied with a felt-tip pen or lipstick, defeat face recognition? I have not read anything about this specifically, but I would bet that it can. Facemasks themselves provide good protection from face ID, and COVID-19 is not going away any time soon.

The problem with these techniques (particularly my birthmark suggestion) is that you don’t know what technology is being used for face recognition, and useful adversarial techniques depend highly on the specific face recognition model. The CVDazzle site states clearly that it’s designs have only been tested against one algorithm (and one that is now relatively old.) Juggalo makeup doesn’t alter basic facial structure. Fake birthmarks would depend on very specific vulnerabilities in the face recognition algorithms. Even with facemasks, there has been research on reconstructing images of faces when you only have an image of the ears.

Many vendors (including Adobe and YouTube) have provided tools for blurring faces in photos and videos.  Anyone who is at a demonstration and wants to take photographs should use these.

But we shouldn’t limit ourselves to defense. In many cities, police refused to identify themselves; in Washington DC, an army of federal agents appeared, wearing no identification or insignia. And similar incognito armies have recently appeared in Portland, Oregon and other cities. Face recognition works both ways, and I bet that most of the software you’d need to construct a face recognition platform is open source. Would it be possible to create a tool for identifying violent police officers and bringing them to justice? Indeed, human rights groups are already using AI: there’s an important initiative to use AI to document war crimes in Yemen. If it’s difficult or impossible to limit the use of facial recognition by those in power, the answer may well be to give these tools to the public to increase accountability–much as David Brin suggested many years ago in his prescient book about privacy, The Transparent Society.

Technology “solutionism” won’t solve the problem of abuse—whether that’s abuse of technology itself, or more plain old physical abuse. But we shouldn’t naively think that regulation will put technology back into some mythical “box.” Face recognition isn’t going away. That being the case, people interested in justice need to understand it, experiment with ways to deflect it, and perhaps even to start using it.

1. 22 Principles for Great Product Managers — This list—pieced together over the past few years—reflects what I believe are some of the most important principles for product managers.
2. Tempering Expectations of GPT-3 and AGI — When I was curating my generated tweets, I estimated 30-40% of the tweets were usable comedically, a massive improvement over the 5-10% usability from my GPT-2 tweet generation. However, a 30-40% success rate implies a 60-70% failure rate, which is patently unsuitable for a production application. (via Simon Willison)
3. Stalk Studio — open source experimental debugger & profiler built on top of distributed tracing. Intuitive DevTools-like UI for extracting useful information from complex traces; Visualize & inspect multiple traces on the same stage; It supports Jaeger and Zipkin out of the box. Nifty!
4. Discovering Symbolic Models from Deep Learning with Inductive Biases — We develop a general approach to distill symbolic representations of a learned deep model by introducing strong inductive biases. We focus on Graph Neural Networks (GNNs). The technique works as follows: we first encourage sparse latent representations when we train a GNN in a supervised setting, then we apply symbolic regression to components of the learned model to extract explicit physical relations. We find the correct known equations, including force laws and Hamiltonians, can be extracted from the neural network.

1. A Bug-Sized Camera for Bug-Sized Robots and Bug-Sized Bugs — With a focus on small size and efficiency, they started with an off-the-shelf ultra low-power image sensor that’s 2.3 mm wide and weighs 6.7 mg. They stuck on a Bluetooth 5.0 chip (3 mm wide, 6.8 mg), and had a fun time connecting those two things together without any intermediary hardware to broadcast the camera output. A functional wireless camera also requires a lens (20 mg) and an antenna, which is just 5 mm of wire. An accelerometer is useful so that insect motion can be used to trigger the camera, minimizing the redundant frames that you’d get from a robot or an insect taking a nap. An astonishing piece of engineering, but also a disturbing advance against privacy.
2. AI Writing Code Makes Software Engineers More Valuable — tools do not lead to less need for Software Engineers, because there is not a fixed limit of demand for software to be built. Rather, they allow more and better software to be built faster, opening new opportunities to apply software solutions where they weren’t previously economical, growing the software industry and actually increasing the value of the skillset needed to build software, which again, is not the same thing as the ability to write code. It’s good to see developers starting to engage with the idea of AI code generation. They have to be careful not to fall into the “only humans can …” trap, which becomes the secular version of “God of the gaps”. Only people can code — oh, code can code? Well, only humans can interpret requirements. Oh, code can translate requirements to code? Well, only humans can …
3. Arguman — an [open source] argument analysis and mapping platform. I’m a huge fan of software that tries to amplify the better angels of our nature.
4. Pipeline Patterns for Continuous Delivery — (1) Pipeline logic is codified, stored alongside application or infrastructure code and utilizes containerized runners. (2) Reusable libraries contain common pipeline logic that is referenceable from pipeline code and independently developed and tested. (3) Build and deploy pipelines should be logically separated, independently runnable and triggered by automated or manual events. (4) Branch commits, pull requests, and merges to the mainline can all trigger different pipeline behavior, optimized to the team’s way of working. (5) Every commit automatically triggers the right pipeline, with build pipelines especially optimized for speed and quick reporting of any issues. (6) Only versioned packages produced by the build pipeline are deployed and these deployments are triggered by humans or automated events. (7) Deploy tagged releases to production and automate the paperwork but leave a paper trail.

1. Matters Computational: Ideas, Algorithms, Source Code — Interesting book about how to do things at the low level. This is a book for the computationalist, whether a working programmer or anyone interested in methods of computation. The focus is on material that does not usually appear in textbooks on algorithms. It’s the kind of book where “word” means a series of bits not a series of characters.
2. Hijacking Control Flow in Web Assembly — The more that WASM takes off, the more that programmers will need to know these kinds of weaknesses. For more on WASM security, see this IEEE Spectrum article.
3. Microsoft Open-Sources Haystack — At a high-level, Hyperspace offers users the ability to: (1) Build indexes on your data (e.g., CSV, JSON, Parquet). (2) Maintain the indexes through a multi-user concurrency model.(3) Leverage these indexes automatically, within your Spark workloads, without any changes to your application code for query/workload acceleration.
   
   When running test queries derived from industry-standard TPC benchmarks (Test-H and Test-DS) over 1 TB of Parquet data, we have seen Hyperspace deliver up to 11x acceleration in query performance for individual queries. Code on Github.
4. Shoelace — A forward-thinking library of web components. Works with all frameworks; Works with CDNs; Fully customizable with CSS; Open source. Clever approach: only contains components that work on all browsers, so if you colour within these lines then your creation will work on all browsers.

1. Pseudogen — [open source] tool to automatically generate pseudo-code from source code. Implements Learning to Generate Pseudo-code from Source Code using Statistical Machine Translation and is written up in Pseudogen: A Tool to Automatically Generate Pseudo-code from Source Code. Exciting work for people inheriting codebases, and perhaps the first step in automated reasoning about code functionality (understand what it does, rewrite it better).
2. Splitgraph — Splitgraph allows the user to manipulate data images (snapshots of SQL tables at a given point in time) as if they were code repositories by versioning, pushing and pulling them. It brings the best parts of Git and Docker, tools well-known and loved by developers, to data science and data engineering, and allows users to build and manipulate datasets directly on their database using familiar commands and paradigms. Open source.
3. Escape Room in Google Forms — OnOn the day the Peters Township Public Library in McMurray, Pennsylvania, was supposed to unveil a superhero-themed escape room, the library had to close its doors due to the coronavirus pandemic. With no physical location to work with, librarian Sydney Krawiec started to devise an alternative: a digital escape room created in Google Forms. In the space of four hours, she made a Harry Potter-themed game that sent participants through a series of challenges based on locations from the book series, and they had to find their way out by solving puzzles. The Google Form went viral. And after other librarians saw it, they decided to make their own.
4. Description to Design — (Twitter) Clever! A description of the webpage is used to generate the page. It’ll be interesting to see how complex it can get, and how well it generates. Trained from github data, which means its design sense is as good as the average of the projects on github … gulp.

Microservices seem to be everywhere. Scratch that: talk about microservices seems to be everywhere.

And that’s the problem. Thinkers as dissimilar as Plato, Robert Boyle, and Keith Richards tend to agree about one thing: Talk is cheap. So we wanted to determine to what extent, and how, O’Reilly subscribers are empirically using microservices. In other words, how long have people been using them? What are they using them for? Are they having success? If so, what kinds of benefits are they seeing? What can we learn from their failures?

So we did what we usually do: we ran a survey. The survey ran from January 31, 2020 through February 29; we had 1502 respondents from the readers of our mailing lists. Here’s a summary of our key findings:

Most adopters are successful with microservices.

A minority (under 10%) reports “complete success,” but a clear majority (54%) describes their use as at least “mostly successful” and 92% of the respondents had at least some success.

Microservices practices are surprisingly mature

About 28% of respondents say their organizations have been using microservices for at least three years; more than three-fifths (61%) of the respondents have been using microservices for a year or more.

Adopters are betting big on microservices

Almost one-third (29%) of respondents say their employers are migrating or implementing a majority of their systems (over 50%) using microservices.

Success with microservices means owning the software lifecycle

Most (74%) respondents say their teams own the build-test-deploy-maintain phases of the software lifecycle. Teams that own the lifecycle succeed at a rate 18% higher than those that don’t.

Success with containers

Respondents who used containers to deploy microservices were significantly more likely to report success than those who didn’t. For our audience, use of containers is one of the strongest predictors of success with microservices.

It’s the culture

A plurality of adopters cite cultural or mindset barriers to adoption. Decomposing monolithic applications into microservices is also a major challenge, but complexity may be the biggest challenge of all.

Respondent Demographics

Technical roles dominate, but management roles are represented, too. Software engineers comprise the survey audience’s single largest cluster, over one quarter (27%) of respondents (Figure 1). If you combine the different architectural roles—i.e., software and systems architects, technical leads—architects represent almost 28% of the sample. Adding architects and engineers, we see that roughly 55% of the respondents are directly involved in software development. Respondents in management roles constitute close to 23% of the audience; executive-level positions (vice presidents and CxOs) account for roughly 13%, overall.

That said, the audience for this survey—like those of almost all Radar surveys—is disproportionately technical. Technical roles represented in the “Other” category include IT managers, data engineers, DevOps practitioners, data scientists, systems engineers, and systems administrators. Each of these roles comprised less than 3% of all respondents.

Figure 1: Respondent roles

Respondents work in more than 25 different vertical industries.The single largest cluster consists of respondents (26%) from the software industry. Combined, technology verticals—software, computers/hardware, and telecommunications—account for about 35% of the audience (Figure 2).

Figure 2: Respondent industries

The second-largest cluster (exclusive of “Other”) was finance, at close to 12%, followed by consulting/professional services, at roughly 10%. The “Other” category, at more than 22% of the total, includes respondents from education (3%, combining K12 and higher education), insurance (3%), energy and utilities (2%), media/entertainment (2%), non-profit (2%), consumer products (1%) and other verticals.

Microservices aren’t just for the big guys. Large companies are amply represented, and respondents affiliated with organizations of 1,000 or more employees comprise 42% of the audience. Overall, however, small- and medium-sized organizations predominate. The largest single cluster (33% of all respondents) was organizations with under 100 employees.

A good global mix. We’re used to seeing a North American-centric tilt in our Radar surveys. This is true, too, of the audience for this survey, nearly half of which (over 49%) works in North America. Europe—inclusive of the United Kingdom—was the next largest region, comprising almost 26% of all respondents. Asia-Pacific accounts for roughly 15% of respondents. In total, just over half of the audience hails from outside of North America. This roughly mirrors usage on the O’Reilly learning platform.

Microservices Adoption: Explication and Analysis

Slightly more than one-third of respondents say their organizations have been using microservices for between 1 and 3 years (Figure 3). This is the single largest cluster in our sample. The second-largest cluster, at nearly one-quarter (over 23%) of respondent organizations, does not use microservices at all. Mature adopters, companies that first adopted microservices between 3 and 5 years ago, comprise the third-largest cluster, at 18%. If you factor in the respondents whose organizations first adopted microservices over 5 years ago, this means that more than 28% of respondent organizations have been using microservices for at least three years. A little under one-sixth of companies (15%) are just beginning microservice adoption, however; they’ve been using microservices for less than 1 year.

Figure 3: Duration of use of microservices

The pace of adoption, in which a clear majority (61%) of respondents say their organizations have used microservices for between 1 and 5 years, roughly tracks with empirical trends we’ve observed internally (e.g., with respect to keyword search and topic usage activity on the O’Reilly online learning platform) and in other contexts. The share of adoption for the 12-month period between January of 2019 and January of 2020 likewise suggests that microservices remains a topic of interest to O’Reilly users. On the other hand, there’s evidence from other quarters that interest in microservices—both as a topic and as a general search term—has flattened (and perhaps even cooled) over the 12 months.

So not only are most respondents using microservices, but three-fifths (61%) have been using them for a year or more. But do they describe their experience as successful? What share of the systems they’re deploying or maintaining are built to microservices architecture? What characteristics do successful adopters share? What can we learn from those who say they’ve failed with microservices?

Critical Factors for Success

A majority of respondents (55%) say their organization’s use of microservices has been either a “complete success” (close to 9%) or “mostly successful” (Figure 4).

Figure 4: Success with microservices

More than one-third (37%) say they’ve had “some success” with microservices. Approximately 8% say they haven’t been successful at all. However, relationships in the data hint at several critical factors for success. For example, in organizations in which development teams own the entire development cycle (i.e., building, testing, deployment, and maintenance), close to half (almost 49%) also reported being “mostly successful” with microservices—and more than 10% said their microservices development efforts were a “complete success. (Figure 5). The combined tally (close to 59%) is about 18% higher than the baseline tally for the survey audience as a whole.

Figure 5: Do dev teams own/not own the software lifecycle?

Do we see meaningful connections between success with microservices and the use, or disuse, of specific technologies? Perhaps; we’ll take a look at that next, specifically with respect to containers, centrally managed databases, and monolithic UIs. The semantics get a bit twisted, but the connections we identify seem more strongly associated with “disuse” than with “use.”

We emphasize the qualifier “seem” because—with respect to the problem of identifying factors for success—we lack detailed data to draw rigorous conclusions. We kept our survey simple; our priority was to develop a general sense of how, why, and in which scenarios people are using microservices. For the most part, our questions didn’t drill down into specifics. We emphasize “seem” for another reason, too. When we asked respondents to cite the biggest challenges to microservices adoption, two problems—decomposition and complexity—stood out for us. Decomposition because it came in at #2, trailing only corporate culture / mindset, which (in almost all Radar surveys) is #1; complexity because two ostensibly different kinds of complexity—“increased complexity” (#4) and the “complexity of managing many services” (#5)—cracked the top 5. Add them together and respondents view complexity in one form or another as the biggest challenge.

The upshot is that complexity in general and (more specifically) the complexity associated with decomposition are the shoals in which most microservices projects seem to run aground. That’s hard to argue with; complexity is rarely (if ever) associated with success. But it’s easy to pretend that complexity is the enemy when it’s really only one variable in a set of tradeoffs.

What does that mean? Although it’s not a question we asked, anecdotally we hear that most microservices projects are replacing large, legacy, monolithic software systems. Those monoliths are themselves very complex, having evolved over decades. The complexity of the monolith that’s being replaced is a “sunk cost” that has only partially been paid; it continues to extract a toll as that software is extended to support new features, changing business models, changing modes of user interaction, and more. Microservices may require paying a complexity cost again, but that’s where the tradeoff comes in: in return for the complexity of re-engineering the system, you get increased flexibility, including a simpler path to adding new features, simpler management, simplified scaling as the organization grows.

Use of Containers for Microservice Deployment

We asked respondents what proportion of their microservices they deploy using containers.

The largest single cluster (38%) for this question consists of respondents that do not deploy their microservices in containers. One might reasonably expect that containers would be the most common means of instantiating microservices. This just isn’t the case, however. Our findings indicate that, for our survey audience as a whole, most respondents (58%) deploy microservices using some medium other than containers (Figure 6).

Figure 6: Proportion of microservices deployed using containers

There are valid reasons not to use containers. For some adopters, technical debt (in the form of custom-built, proprietary and monolithic systems, applications, or architectures) is a constraining factor. So it just makes sense to instantiate microservices at the level of the virtual machine (VM), as distinct to that of the container. Or maybe it’s faster and less costly, at least in the short term, to build and instantiate microservices as non-virtualized code running in the context of a conventional operating system, a database, application server, etc..

This doesn’t mean respondents aren’t using containers; most of the survey audience is. It’s just that containers are not yet the most popular means of instantiating microservices, although that could be changing. For example, the second-largest cluster (31%) for this question consists of organizations that deploy between 75% and 100% of their microservices using containers. And 11% use containers to deploy between 50 and 75% of their microservices.

The upshot is that more than two-fifths (42%) of respondent organizations use containers to deploy at least half of their microservices—and that, for the survey audience as a whole, nearly two-thirds (62%) are using containers to deploy at least some of their microservices. So we have a split: on the one hand, most microservices are instantiated using a technique other than containers; on the other hand, most organizations that use microservices also instantiate at least some of them in containersFor example, 10% of respondents say they use containers to deploy between 10-25% of their microservices; a little more than 9% deploy between 25-50% of microservices with containers; and, again, 11% deploy between 50-75% using containers.1 It seems adopters either go (mostly) all-in on containers, using them for most microservices, or use them sparingly.

There’s a critical, and intriguing, “but” here: a higher than average proportion of respondents who report success with microservices opt to instantiate them using containers; conversely, a much higher proportion of respondents who describe their microservices efforts as “Not successful at all” do not instantiate them in containers. For example, almost half (49%) of respondents who describe their deployments as “a complete success” also instantiate most of their microservices (75-100%) in containers. This is more than 5x the baseline (9%) for this question. Conversely, an overwhelming majority (83%) of respondents who describe their microservices efforts as “Not successful at all” are instantiating them by some means other than containers. (These are respondents who use containers in conjunction with less than 10% of their microservices.) This is about 11x the baseline for this question (Figure 4).

This observation makes intuitive sense. With microservices, instead of deploying one monolithic application, you may need to deploy and manage hundreds or thousands of services. Using containers to standardize deployment, and container orchestration tools to automate ongoing management, greatly simplifies the burden of deployment and management. It’s worth remembering that “containers” get their name by analogy to shipping containers: instead of loading a ship with 10,000 cases of canned soup, 20,000 board-feet of lumber, and a few thousand automobile tires, the shippers would pack their goods in standardized containers that could be stacked up on the ship. This represents a significant reduction in the cost of shipping–or the operational cost of deploying and maintaining software. Containers are a simplifying technology.

Use of a Central, Managed Database

When we asked respondents what proportion of their microservices share a central, managed database (Figure 7), we found that not using a centrally managed database with microservices tends to be associated with failure. That said, the question itself (i.e., “What percentage of your microservices share a centrally-managed database?”) doesn’t give us much to go on. We don’t know what respondents considered a centrally managed database; what kind of database they were using (relational or non-relational); or whether transactional integrity was an issue.

Figure 7: Proportion of microservices that use a centrally managed database

Granted that there’s a lot we’d like to know that we don’t, let’s think about why this question is important, and dig a bit deeper. The point of microservices is breaking an application into separate, decoupled services. Using the database as the integration point between services is counter to this purpose. The result is a software system that looks like it’s built from microservices, but really isn’t, and which realizes few of the promised advantages because the individual services are still coupled to each other at the database layer.

Are our respondents succeeding with decoupled databases? For our survey audience, it looks like the smaller the proportion of microservices that share access to a central, managed database, the greater the likelihood of failure. Think of this as what survivorship bias is most apt to miss: i.e., failure. The lesson in such cases is usually that failure tells a valuable story of its own.

Of the respondents who said that they are “not successful at all,” 71% said that they did not make much use of a centrally managed database (under 10% of their microservices). The problem is that we don’t actually know what using (or not using) a centrally managed database entails because the question itself is imprecise. For example, a recommended practice is to implement a separate namespace for each microservice. This can be accomplished in several ways—including by using a centralized database!2 But implementing a separate namespace for each service raises a number of hard problems, starting with transactional integrity and data consistency, and (notionally) encompassing security and privacy, too. Working through these issues isn’t easy, even with a centralized database. It’s much harder without one, however.

However, while it’s easy to look at these failures and suppose that using a separate database (or database schema) per service is a bad idea that leads to failure, that’s far from the whole picture. Think back to where we started this section: microservices represents a tradeoff of complexity against flexibility. (And also a tradeoff of complexity now versus complexity accreted over the years in a legacy system–we could perhaps think of paying off technical debt with a big “balloon payment.”) Replacing a legacy monolith with microservices isn’t easy. But working through the current complexity to build a system that’s more likely to serve your future needs is what good engineering is all about. And when you’re building a complex system, you need to know where the pain points are. That’s what this data is really telling us: redesigning databases to eliminate dependencies between services is a significant pain point.

Monolithic UI

We also asked users what percentage of the systems deployed as microservices had a monolithic user interface (Figure 8). This question is also problematic. First, what is a monolithic UI? More to the point, what would our respondents assume we mean by a monolithic UI, and what are they doing if they’re not building a monolithic UI? There are several possibilities, none of which is entirely convincing. One recent trend in web development is “micro frontends”, but this doesn’t appear to be well-known enough to have a significant effect on our data. Respondents are likely to associate a “monolithic UI” with traditional web development (or perhaps even desktop applications or ancient “green screens”), as opposed to the use of components within a framework like React. (Since micro frontends are built with frameworks like React, these may be two names for almost the same thing. Once you’re using a component-based framework, associating components with individual back-end services isn’t a big leap.) In any case, it is likely that a monolithic UI represents a tradeoff for simplicity over flexibility and complexity. A front end built with a modern reactive web framework is almost certainly more flexible–and more complex–than a 1990s-era web form.

Use of a monolithic UI is another possible example of survivorship bias; it’s also another case in which respondents who are successful with microservices are much less interesting than those who aren’t.

Figure 8: Proportion of microservices that use a monolithic UI

Take the cluster of respondents who don’t build a monolithic UI. (That is, <10% of their microservices use a monolithic UI.) At 31% of all respondents, this is the largest group. But it also has the highest rate of complete failure: 17% describe their implementations as “Not successful at all.” Respondents in this group were also more likely to describe their projects as a “complete success” (12%); 37% said they were “mostly successful.”

At the other end of this spectrum, among respondents who used a monolithic UI for 50-75% of their systems, only 2% said they were not successful at all. 8% said their projects were a “complete success,” while 52% said their projects were “mostly successful.”

What are we to make of this? It’s hard to say, given that the notion of a monolithic UI (and its opposite) are poorly defined. But this looks similar to what we observed for databases. Reactive web frameworks (like React and Angular), which are used to build many modern web interfaces, are very complex systems in their own right. It’s not surprising that systems that don’t use a monolithic UI have a high failure rate, and that systems that implement a monolithic UI have a fairly high success rate; if simplicity for developers were the only criteria, we’d all go back to green screens.

Rather than just looking at a “complexity tax,” we have to consider what the goal is: building systems that can easily be extended and scaled as conditions change. Is a UI that isn’t monolithic (regardless of the technology) more complex? Probably so. Ideally, moving a monolithic application’s back end to microservices while maintaining the legacy front-end would be minimal work (though “minimal” is always more minimal in theory than in practice). What’s gained in return? Additional flexibility; in this case, we’d be looking for the ability to support different kinds of user interfaces and different kind of interactions. Adding a mobile or voice UI to a legacy front end can easily become a nightmare; but it’s hard to imagine any modern application that doesn’t have a mobile front-end, and in a few years, it will be hard to imagine an application without a voice-driven front end. Re-envisioning the front end in terms of services (however they may be implemented) is building flexibility into the system. What other user interfaces will we need in the next few years? Again, the issue is never simply “complexity”; it’s what you get in return.

Benefits and Challenges

Respondents were asked which benefits, if any, they attribute to their successful use of microservices (Figure 9). They were asked to select all applicable benefits.

The largest cluster for this response (at 45%) named “feature flexibility,” followed (at just under 45%) by “responding quickly to changing technology and business requirements.” Just under 44% cited the benefit of “better overall scalability,” followed (43%) by “more frequent code refreshes.” The least-cited benefit (15%) was that of lower development costs; just 20% of respondents say they realized one of the core promises of microservices: improved availability by virtue of multiple, redundant functions (i.e., services).

Figure 9: Benefits of using microservices

Almost all of our Radar surveys have found that respondents cite corporate culture as one of the top two or three impediments to adoption or success. This survey was no exception. Culture was the most oft-cited challenge: almost 40% of respondents cited culture—or, alternatively, the problem of “overcoming existing mindset”—as among the biggest challenges they faced in adopting microservices (Figure 10). This was the largest cluster for this response, followed (37%) by the challenge of decomposing requirements into primitive/granular functions. The challenge of integrating with legacy systems was third, cited by approximately 30% of respondents.

If we combine the two responses that have to do with complexity (“Increased complexity” and “Complexity of managing many services”), we find that complexity in one form or another is the biggest overall challenge, cited by 56% of respondents.

Figure 10: Challenges encountered in adoption of microservices

Other results of note: at No. 6, technical debt (cited by 28% of respondents) just missed the top five. It’s the kind of hard problem that cannot be wished, willed, or architected away–indeed, replacing a monolithic system with microservices can be seen as a “lump sum” payment of years of accrued technical debt. The rest of the top 10 reads like a laundry list of hard problems: API quality (28%), talent and skill shortages (28%), training and retraining challenges (also 28%) and monitoring/observability (27%) round out the top 10. The high showing for API quality—i.e., their richness, robustness, transparency, accessibility, stability (over time) and usefulness—shouldn’t be surprising. The relatively low result for security (No. 11, cited by 21% of respondents) is surprising.

Takeaways

For the most part, adopters say they’re having success with microservices: more than half (55%) describe their efforts as at least “mostly” successful. Respondents commonly attribute several clear benefits to their use of microservices, including feature flexibility; the ability to respond to changing business requirements; improved scalability; and more frequent code releases. And, at least among adopters, microservices comprise a growing share of production systems: almost half (46%) of respondents say their organizations are currently developing for or migrating one-quarter or more of their production systems to microservices. Nearly one-sixth (15%) say they’re developing or migrating between 75-100% of their systems to a microservices-oriented architecture. For our audience, microservices architecture is not only established but (for a majority of adopters) used to support systems or workloads in production.

There’s evidence here for the microservices skeptic, too. About 8% of would-be adopters describe their experiences with microservices as “not successful at all.” Proponents will claim that 8% is a shockingly low rate of failure for any software development project, particularly given the historical failure rates of large IT projects; but skeptics will counter that 8% is still, well, 8%. And even if a majority of respondents say they’ve been “mostly successful” with microservices, a sizable percentage (37%) say they’ve had only “some” success. The upshot is that a little under half (45%) of all users have had bad, middling, or only modestly successful experiences with microservices. A skeptic might also point to selection bias: not only is our audience dominated by people in technical roles, but software developers and architects are overrepresented. A survey like this is bound to attract a disproportionate share of participants with an interest in seeing microservices succeed—or fail. In the same way, the person who volunteers her time to complete our survey is more likely than not to be working with or to have an interest in (for or against) microservices. These are all valid objections.

Our results emphasize the importance of pragmatic microservices development. Microservices can be complex–there’s no point in denying that. And complexity frequently leads to failure–there’s no point in denying that, either. Using containers for deployment is a way of minimizing complexity. We’ve seen that decoupling databases is frequently a pain point, along with redesigning the user interface as a set of components. But again, it’s important to recognize that microservices rarely arise out of nowhere. Many microservice projects are replacing existing systems. And that existing system has its own complexity: complexity that grew over the years, complexity that’s preventing you from achieving your current goals. If you’ve become accustomed to your legacy systems, you probably don’t realize how complex they are–but if you’ve made the decision to migrate from a monolith to microservices, the complexity of maintaining your monolith is almost certainly the reason.

Replacing one kind of complexity with another–is that a gain? Absolutely, if it’s done correctly and enables you to achieve your goals. Complexity is an engineering problem, and engineering problems are always about tradeoffs. When you’re building microservices, keep your eye on the goal–whether that’s supporting new features, scaling for more customers, providing a new experience for users, or something else. Don’t be surprised that microservices bring their own complexity, and don’t let that daunt you. But don’t underestimate the challenge, either.

Special thanks to Kristen Haring, Phil Harvey, Mark Madsen, Roger Magoulas, Larry “Catfish” Murdock, Sam Newman, and Mac Slocum for—among other invaluable contributions—questions, comments, context, and, above all, constructive criticism.

Footnotes

1 For example, 10% of respondents say they use containers to deploy between 10-25% of their microservices; a little more than 9% deploy between 25-50% of microservices with containers; and, again, 11% deploy between 50-75% using containers.

2 A development team could use a centrally managed database to create separate namespaces for each service. This practice is notionally consistent with the idea that each microservice should own its data. It also saves developers the headache of coding logic to manage data consistency, transactional durability, and other issues. Or, alternatively, software developers could opt to manage data persistence (along with data consistency and transactional durability) on a per-microservices basis, a practice that is strictly consistent with the idea of data sovereignty, but which radically ups the complexity factor.

1. Fixing Bugs Properly — Why did a fix that seems so simple when looking at the changes made take two days to complete? Good programmers leave things better than they found them.
2. Data Structures and Algorithms I Actually Use — This article is a set of real-world examples where data structures like trees, graphs, and various algorithms were used in production. It’s good to see them used outside interviews.
3. Trillian — Trillian implements a Merkle tree [cryptographically-verified tree] whose contents are served from a data storage layer, to allow scalability to extremely large trees. On top of this Merkle tree, Trillian provides two modes: An append-only Log mode […], An experimental Map mode that allows transparent storage of arbitrary key:value pairs derived from the contents of a source Log.
4. LeakyPick: IoT Audio Spy Detector — Our proof-of-concept is a LeakyPick device that is placed in a user’s smart home and periodically “probes” other devices in its environment and monitors the subsequent network traffic for statistical patterns that indicate audio transmission. Our prototype is built on a Raspberry Pi for less than USD40 and has a measurement accuracy of 94% in detecting audio transmissions for a collection of 8 devices with voice assistant capabilities. Furthermore, we used LeakyPick to identify 89 words that an Amazon Echo Dot misinterprets as its wake-word, resulting in unexpected audio transmission.

A World of Individuals

Our world is on fire. For too long we have been complicit, maintaining the status quo. This must change. I’m writing this as we’re living through three pandemics. Covid-19, structurally enabled racism, and the climate crisis. Each of these pandemics make visible the ways that individuals fit into and contribute to wider communities. Or fail to fit in.

Before Covid-19, it’s likely that society made you think of liberal politics or charity or posh events that you rarely get invited to. Now, I bet you think of society as community. We’ve all become more aware of how connected we are to one another, and our responsibilities to our communities. That’s not only for us as people, but for business too.

Until now innovation frameworks like human-centered design, jobs to be done or design thinking have focused on two things. The first is serving the individual in order to make products and services that people want. This is often called “user needs.” The second is following a “growth imperative,” and a financial measure of success. We’ve asked how software and hardware could help drive profit and GDP.

The result is products and services that give us superpowers. You have a pocket supercomputer that can conveniently beckon a taxi instantly to you. You know exactly where you are. You can instantly communicate and transact from anywhere. However these superpowers come at a cost: your attention is a scarce commodity and your data is put to use in an ocean of advertising.

What strikes me the most is that these innovation frameworks only work when organizations and their teams operate from a place of privilege. It’s easier to ignore the unintended consequences (the “negative externalities” in the industry’s dry jargon) when you are personally not impacted. So it’s unsurprising that institutions like Harvard or Stanford gave birth to these methodologies, Silicon Valley companies popularized them, and consultancies packaged them into toolkits. These are organizations that are often predominantly white, male, and affluent.

Here’s an example. A new generation of electric bike and scooter rideshare startups made it possible to pick up or leave a scooter anywhere in a city. They removed the charging docks, so the user no longer had to find one that had a scooter available to be used, or had a spare slot to return the scooter to. The user would just tell the app that they’d finished with it, and the app took care of the rest, which was very convenient. But this made the experience of moving around the city harder for everyone else, and particularly affected those with limited mobility. So much focus went into serving the needs of the individual that the needs of other people went ignored.

It doesn’t stop at products and services. Individualism has shaped much of our social contract in the West. Data protection frameworks like Europe’s GDPR give us individual rights to data that represents us. Consent is asked for on an individual basis, with cookie banners being the ultimate divide-and-conquer attack on humanity. If consent was gained through hurried and weary clicks on nagging pop-up barriers on countless websites, can it be considered freely given?

A problem with design for the individual is the assumption that data necessarily represents one person. But it doesn’t. In nearly all cases, data represents many people. For example your DNA represents your parents, siblings, and children. Your location data represents where you’ve been with other people, where you see your friends or the walk to school with your kids. Data is inherently social and represents society.

Society-Centered Design

This is where Society-centered design comes in. At IF, a technology studio that I run, we believe it’s time to do better. It’s time to make better design approaches and tools, better measures of success, and better data protection standards. We need a new framework for products, services, and data that is purpose-built for the 21st century. We want to move beyond human-centered design to society-centered design. Where society, not an individual, is at the center.

Society-centered design is about changing the climate of ideas, and moving towards real-world solutions. It’s an approach to problem solving, that develops solutions by putting society in every step of the problem-solving process.

How could we make rideshare scooters work for wider society as well as the riders? This question is timely because the UK Government is accelerating a plan to allow rentable electric scooters in towns and cities to provide socially-distanced alternatives to public transport.  Perhaps the scooters could be dockable so the streets could be less cluttered: the scooter companies would work with local authorities to provide many more docks. The docks could be sufficiently standardized that any scooter or bike could use them for secure storage and charging. Working with open standards, working with the city, working with people.

A different and speculative example is how we might rethink data management. I’ve written before about how hard it is to stay on top of all the consent choices you’ve made. If your data preferences change, are you going to visit a million websites to modify the communication and cookie preferences you clicked half a decade ago? What if instead you were able to delegate decision making to an entity that represents your values? That way the consent management is dealt with by a third party. This isn’t a new idea. Tom Steinberg has written about organizations, “personal data representatives,” that could take on this role. I explored the idea of data cooperatives in 2014. Now there are a growing number of ‘data trust’ experiments from the Open Data Institute, Uber drivers, Sidewalk Labs, and others.

Designing for society means designing for the broader context of systems that we impact and shape. To do this, we must be intentional about citizen empowerment, civic commons, public health, equity, and the planet:

• Citizen empowerment: how might we give people more rights and capabilities?
• Civic commons: how might we create shared resources that strengthen communities?
• Public health: how might we protect the safety and improve the physical and mental health of communities?
• Equity: how might we design products, services, and standards that are fair to everyone, not just the most privileged?
• The planet: how might we better care for our world?

We have a collective opportunity right now to design out the structural inequalities around us. To collectively hold each other accountable, to examine our existing products and services to make sure they are equitable. It’s not enough to play lip service about change within your business. It’s not enough to just look at recruitment practices. It’s not enough to make tweaks to the images you show. To take a society-centered approach means fundamentally looking at the underlying values of your business.

Until now, individual needs have been the foundation of a business’ growth, profit, and culture. Now it’s time to look towards society’s needs.

1. Neutralizing Misinformation through Inoculation — We found that false-balance media coverage (giving contrarian views equal voice with climate scientists) lowered perceived consensus overall[..]. Likewise, misinformation that confuses people about the level of scientific agreement […] had a polarizing effect […]. However, we found that inoculating messages that (1) explain the flawed argumentation technique used in the misinformation or that (2) highlight the scientific consensus on climate change were effective in neutralizing those adverse effects of misinformation.
2. Datascript — An immutable in-memory database and Datalog query engine in Clojure and ClojureScript. Datalog fascinates me — a declarative query language that’s a subset of Prolog.
3. WARduino — In this paper we investigate the feasibility of using WebAssembly to program Arduino compatible microcontrollers. Our experiments lead to extending the standard WebAssembly VM with: 1) safe live code updates for functions and data 2) remote debugging support at the VM level 3) programmer configurable (Arduino) modules in order to keep the virtual machine’s footprint as small as possible. The resulting WARDuino VM enables the programmer to have better performance than an interpreted approach while simultaneously increasing the ease of development. To evaluate our approach, we implemented a simple breakout game and conducted micro benchmarks which show that the VM runs approximately 5 times faster than Espruino, a popular JavaScript interpreter for the ESP32 microcontroller. This is clever! (Source code available)
4. Mobile Manipulators — (IEEE Spectrum) Stretch has been relentlessly optimized to be the absolutely minimum robot to do mobile manipulation in a home or workplace environment.

1. Too Efficiency — (Tim Bray) Think of efficiency as optimising for one variable. As anyone who has seen a greyhound or a dalmation up close knows, optimising for one aspect of a complex system can weaken other aspects. Tim cites software security, food, work hours, and others. The best optimisation advice I received was “never set one target, always at least two: what you hope to get, and what you don’t want to lose to get there.”
2. Voice Separation — (Facebook AI) a new method to separate up to five voices speaking simultaneously on a single microphone. Our method surpasses previous state-of-the-art performance on several speech source separation benchmarks, including ones with challenging noise and reverberations. One step closer to being able to drop a mic into a room and emerge with a transcript of what was said in a meeting.
3. Design Docs at Google — A nice structure which may inspire you when compared to your current design documentation. (You have design documentation, right?)
4. Counterfeit Hardware — We bought over 1000 “waterproof” probes or bare chips from more than 70 different vendors on ebay, AliExpress, and online stores -big and small- in 2019. All of the probes bought on ebay and AliExpress contained counterfeit DS18B20 sensors, and almost all sensors bought on those two sites were counterfeit. Supply chains are hard, says counterfeit Barbiee[tm].

1. Notes on Splicing CRDTs for Structured Hypertext — It’s a great feeling, seeing everyone around you create beautiful 3D graphical masterpieces or fully complete web applications while you spend twelve hours a day struggling to read technical papers that you don’t even understand. This is what happens when you attempt to do things with CRDTs.
2. FinFisher Exposed — FinFisher is not afraid of using all kinds of tricks, ranging from junk instructions and “spaghetti code” to multiple layers of virtual machines and several known and lesser-known anti-debug and defensive measures. What a cunning piece of code this malware is! It has its own virtual machine, which means that typical reverse-engineering tools fail.
3. GDPR Explained by People Who Attack It For Having What it Actually Has — Richard Stallman argues [against the GDPR] that laws must prohibit data collection if it is not necessary or not justified, and that systems must be designed not to collect certain data. That is why, since 1995, EU data protection law regulates not only data use, but also the collection of personal data.
4. Evidence-Based Software Engineering Based on Publicly-Available Data — the first half discusses the major areas of software engineering, driven by an analysis of the publicly available data. The aim is to provide the information needed to reduce the resources needed to build and maintain software systems, and to make efficient use of available resources. Many topics usually covered in software engineering textbooks are not discussed because public data relating to them could not be located. I don’t know that it has that much to say on software engineering, but it’s a fascinating summary/tour of a whole lot of research in many interesting fields.

At Microsoft’s Build conference, Microsoft CTO Kevin Scott talked about an experimental project in which an AI, trained on code in GitHub, actually creates programs: it generates function bodies based on a descriptive comment and a message signature. (Skip to 29:00 of the video.)  Along similar lines, an unrelated research paper reports on unsupervised translation of programs from one language to another–indeed, probably an easier task than natural language translation.

Microsoft’s demo is, well, a demo, and the research paper is a research paper. Microsoft’s AI was able to provide method bodies for some relatively simple functions; it was obviously rather limited. It begs the question of exactly how you write the comment that tells the AI what to do; are we just replacing a description in a precise programming language with a description in an ambiguous, imprecise human language? I’m sure that training the model costs more than hiring a developer to write a few short functions. And while it’s easy for me to believe that software can translate some simple code from COBOL to Rust, I can only imagine that it would be horribly confused by the multitude of tricks programmers used in the 1960s and 1970s to deal with computers that had, by modern standards, miniscule memories. Programming jobs are not going to disappear tomorrow.

Nevertheless, it’s important for us to think about what automation might mean for the future of programming.  Is this a glimpse of the future, and if so, what is it telling us? What can we learn from it? Programming will increasingly be automated; and, as someone who got started writing assembler on a PDP-8, I can tell you that programming is already highly automated, and that a good optimizing compiler is already an advanced AI system that takes your hints and turns them into working code.

Programming is not going to “go away” or “become obsolete.” However, its meaning will change. I’ve previously written about blue- and white-collar programmers: programmers who connect things, and programmers who design the things that are connected, and build the tools to connect them. These are different (though highly overlapping) skillsets. And while Microsoft’s demo may show that programmers may eventually be liberated from the task of coding up simple functions, the code-generating engine was certainly built by a team of programmers, and possibly a large one. The programmers who write the higher-level tools, like Microsoft’s coding engine itself, can breathe a sigh of relief.  As can programmers who design APIs, since you still have to provide the function/method signatures.

And the blue-collar programmers who connect things? While this demo could spit out functions, I saw few signs that it could build larger systems out of the functions that it wrote. Although it had the ability to call functions that it had written, it could not put together a large program from a written specification. It might be able to spit out a simple bill, but it certainly couldn’t spit out a complete billing system. A project for future research? No doubt, but that’s probably dozens of years out. Programmers who connect things—the plumbers of the digital world—are also safe.

But this feat still begs the question about what we mean by programming. In the video, Kevin Scott talks about reducing the time programmers spend on dull, repetitive tasks. Yes, that’s what we all say about AI: it will reduce the time spent on dull, repetitive tasks and free more time for creative work. Let’s break that down, though. Most of programming is specifying, in excruciating detail, how to execute some process. That can be dull, it is often repetitive, and it is certainly error-prone. And we really need to think more about what programming ought to be. To use Scott’s word, what are the “creative” aspects of programming?

I’m not sure that “creative” is the right word. Back in the 1960s and 1970s, more programmers were called “analysts.”  That job title is still in use, but it’s not all that common: Glassdoor shows about 10,000 jobs for “programmer analyst,” 44,000 for “programmer,” and 100,000 for “software engineer.” Monster shows about 20,000 for “programmer analyst,” 300,000 for “programmer,” and 170,000 for “software engineer.”

While the job may have differed little from what it is now, let’s think about what “analyst” means. Analysts analyze a problem; they think about what the problem is, and how to solve it effectively. They think about breaking it up into parts. They may even think about whether it should be solved; they might think about how it should be solved, what ethical issues it raises and how those issues should be handled. Can the software be abused? If so, how? What steps can be taken to prevent abuse? And analysts need to think about how people will use the software: what’s the user interface, what’s the user experience, can it be used by people with disabilities?

In O’Reilly’s Software Architecture Fundamentals Superstream, Thoughtworks’ Rebecca Parsons said that “analysts” were essentially doing software architecture: making big picture decisions about what the software should do, and how it should be built. Analysts and architects need to understand the business case. They need to present decisions about software systems to management in business terms.  She used the need to test a backup strategy as an example: rather than telling the CEO about the technical details, just say “This test will show us how long it will take to get back online if the system crashes on the busiest retail day of the year.” That’s what an analyst does.

Our emphasis on writing code, and on measuring productivity by “lines of code,” is myopic. Just look at all the wonderful (and absolutely necessary) tools we have for building, testing, archiving, and deploying code.  But those tools, essential and revolutionary as they are, don’t address the real issue: are we solving the right problems? There are thousands of applications out there that pass every unit test, every integration test, and every acceptance test, but are still awful to use.  And we are learning that there are new dimensions to software that we’ve rarely thought about: are there groups who need to use our software and can’t? Does your software support accessibility?  Can it be used by people who don’t have reliable access to computers and to networks?  The work of an organization like Code for America isn’t technically profound or deep.  What’s radical about a product like GetCalFresh was the act of re-engineering the system to make it usable by the people who need to use it. And we need a lot more analysis like that.

There’s a lot more to programming than just slinging code and writing functions. The most important parts of the job have nothing to do with writing quicksort on the whiteboard in an interview. There’s a lot to think about; and currently, programmers spend too much time rushing code out the door to meet a release date than they spend thinking about the bigger picture. That’s almost always someone else’s job. But whether or not it gets into production, Microsoft’s research gives us an opportunity to think about what programming really means. What’s the real job? What are we really trying to accomplish?

1. Avoiding Fallback in Distributed Systems — In this article, the focus will be on how fallback strategies can cause more problems than they fix. We’ll include examples of where fallback strategies have caused problems at Amazon. Finally, we’ll discuss alternatives to fallback that we use at Amazon.
2. So You Want to Build an Observability Tool — It may not be as catchy as “three pillars”, but in order to claim your tool delivers observability, it should support/deliver the following: Arbitrarily-wide structured raw events; Context persisted through the execution path; Without indexes or schemas; High-cardinality, high-dimensionality; Ordered dimensions for traceability; Client-side dynamic sampling; An exploratory visual interface that lets you slice and dice and combine dimensions; In close to real-time.
3. What Predicts Software Developers Productivity? — Our results suggest that the factors that most strongly correlate with self-rated productivity were non-technical factors, such as job enthusiasm, peer support for new ideas, and receiving useful feedback about job performance. Compared to other knowledge workers, our results also suggest that software developers’ self-rated productivity is more strongly related to task variety and ability to work remotely.
4. Loopy — I love visual simulations, especially interactive ones, as a way of generating deep understanding of a complex situation. This lets you build a model of interactions and levers, and then you play with it as it simulates flow through the system. It’s not complex, and you can probably think of ways to improve it (you can, it’s open source). But what do you understand that you could simulate for your colleagues to better understand?

1. When Data is Messy — I love stories that illustrate the ways machine learning can draw the wrong conclusions. Researchers at the University of Tuebingen trained a neural net to recognize images, and then had it point out which parts of the images were the most important for its decision. When they asked it to highlight the most important pixels for the category “tench” (a kind of fish) it highlighted human fingers on a green background. Because most of the tench pictures the neural net had seen were of people holding the fish as a trophy. (via Simon Willison)
2. Large Scale Experimentation — The internet era has made data-driven decision making easier, faster, and better than ever before. With it come unique challenges and the possibility to rethink how to optimally experiment. We propose a Bayesian setting that implicitly captures the opportunity cost of having multiple interventions to test. Featuring a nifty simulation to help you feel what it is to learn from experiments.
3. CI/CD for Machine Learning — an ecosystem of tools. CML helps you bring your favorite DevOps tools to machine learning. Ops for ML projects is interesting because it brings new problems with no widely-known solutions.
4. Engineer Productivity — (Charity Majors) Some of the hardest and most impactful engineering work will be all but invisible on any set of individual metrics. You want people to trust that their manager will have their backs and value their contributions appropriately at review time, if they simply act in the team’s best interest. You do not want them to waste time gaming the metrics or courting personal political favor.

1. eDEX-ui — a fullscreen, cross-platform terminal emulator and system monitor that looks and feels like a sci-fi computer interface.
2. Announcing Microsoft Freta — Microsoft have a service for detecting evidence of OS and sensor sabotage, such as rootkits and advanced malware, in memory snapshots of live Linux systems.
3. Placeholdifier — Turn any website into a live wireframe.
4. Work Schedules — (1) How does my organization’s scheduling practices affect employee effectiveness and well-being? (2) Can we better align our work schedules with the needs, desires, and personalities of our employees? (3) What are the implications of creating customized schedules or giving employees more control over their schedules? (4) Can we effectively balance the needs and desires of both the organization and employees? With links to research on each concern.

Austin Godber: Python and Kubernetes

Description:
Austin will briefly introduce containerization and Kubernetes. Show an example of a containerized Python application and then show how to interact with Kubernetes through it's API using Python.

You can attend on July 9th at 7pm by visiting: https://lufthans.bigbluemeeting.com/b/plu-yuk-7xx

1. Debubble — I like that people are trying software like this, to bring out our better angels.
2. Parallel Programming in Multicore OCaml — A glimpse at how OCaml is handling parallelism. There’s a lot to learn from the paradigms of “weird languages”.
3. Language Independent Validation Rules — Multi-language supported validation rules. The two languages that matter the most? Javascript and whatever you’re using on the backend.
4. TaBERT — TaBERT is the first model that has been pretrained to learn representations for both natural language sentences and tabular data. These sorts of representations are useful for natural language understanding tasks that involve joint reasoning over natural language sentences and tables. A representative example is semantic parsing over databases, where a natural language question (e.g., “Which country has the highest GDP?”) is mapped to a program executable over database (DB) tables. Applying ML to database systems is interesting.

1. Open Differential Privacy — Open source software from Microsoft and Harvard. (via Microsoft’s announcement).
2. Engineering Resumes — to help those of you looking for a new job in these uncertain times, here are some examples of what accomplishments look like for software engineers. These are oriented towards individual contributors (perhaps I’ll do an engineering managers version next).
3. Evil C — A 29-byte source file that takes 27m to produce a 16GB executable.
4. Platform Adjacency Theory — (Alex Russell) the web thrives or declines to the extent it can accomplish the lion’s share of the things we expect most computers to do. […] Growing a platform’s success requires unlocking use-cases not already serviced. That mean finding needs that combine things your platform is already good at with a small number of missing capabilities. An interesting essay arguing that Apple and Mozilla are underinvesting in web feature development and thus threatening the web metaplatform.