You are here


Why we need to think differently about AI

O'Reilly Radar - Wed, 2018/08/08 - 04:00

General intelligence or creativity can only be properly imagined if we peel away the layers of abstractions.

While AI researchers are racking up ever more impressive successes, in many respects AI seems to be stalled. Significant as the successes are, they’re all around solving specific, well-defined problems. It isn’t surprising that AI or ML can solve such problems. What would be more surprising, and what seems no closer than it was a decade ago, would be steps in the direction of general intelligence or creativity.

How do we get to the next step? When a problem can’t be solved, it’s often because you’re trying to solve the wrong problem. If that’s the situation we find ourselves in, what’s the right problem? How do we set about asking a different set of questions?

I am impressed with Ellen Ullman's critique of abstraction in her book Life in Code. And I am impressed by the way that abstraction, such a critical concept in computing, has permeated our notion of artificial intelligence. One way to summarize the current state of artificial intelligence and machine learning is as a technique for automating abstraction. The myriad hyperparameters in a model represent a particular abstraction, a particular way of compressing the complexity of meatspace into rules that a machine can follow.

Those abstractions have led us to think about AI in inappropriate ways. We congratulate ourselves when we build systems that can identify images with greater accuracy than a human, but we often forget that those systems give incorrect, and possibly even harmful, answers. Unlike humans, they don't know how to say, "I don't know what that is." Yes, humans can misidentify things, too. But we can also say, "I don't know," as Pete Warden has pointed out. And frequently, we do. Whether it represents cluelessness or wonder, “I don’t know” is what abstraction leaves out—at least in part because it’s difficult to represent.

In the Meaningness blog, David Chapman suggests that one problem with AI is that we don't know how to evaluate progress. AI falls into the gap between five disciplines—science, engineering, math, philosophy, and design—plus an ill-defined sixth: spectacle. Briefly, "spectacle" means "gives good demos." Science means developing accurate, predictive models; engineering means developing useful applications of those models.

Chapman points out that we only have very fuzzy ideas about what our own "intelligence" means. We don’t have the abstractions—either as scientific models or pragmatic engineering results, and certainly not as mathematical theorems—to understand intelligence. And since we don't really know what intelligence is, we tend to say, "well, that sort of looks intelligent. We don't know exactly what it's doing or why, but we don't understand our own intelligence, either. So, good enough." And that isn't good enough. That's just good enough to deceive ourselves and, in the process, prevent us from understanding what we can really do. But if we don’t have appropriate abstractions, what else can we do but watch the demos?

So, how do we make progress? After all, our metrics for progress are, in essence, abstractions of abstractions. If our abstractions are the problem, then second-order abstractions to measure our progress are not going to point us in the right direction. I'm making a difficult problem more difficult, rather than less. But also: when you're up against a wall, the most difficult path may be the only one to take. Is there a way to conceive of AI that is not an abstraction engine?

Kathryn Hume, in “Artificial Intelligence and the Fall of Eve,” writes about the concept of the “Great Chain of Being”: the Aristotelian idea that all beings, from rocks up to God, are linked in a hierarchy of intelligence. It's also a hierarchy of abstraction. In Paradise Lost, John Milton's Eve is committing a sin against that order, but in doing so, she's also committing a sin against abstraction. The serpent tempts her to move up on the chain, to usurp a higher level of rationality (and hence abstraction) than she was given.

Hume's radical suggestion is that Eve didn't go far enough: the opportunity isn't to move up in the Great Chain of Being, but to move beyond it. "Why not forget gods and angels and, instead, recognize these abstract precipitates as the byproducts of cognition?" What kinds of AI could we build if we weren't obsessed with abstraction, and could (going back to Ullman) think about the organic, the live, the squishy? From the standpoint of science, engineering, and mathematics (and maybe also philosophy and design), that’s heresy. As it was for Eve. But where will that heresy lead us?

The phrase "Big data is people" has gained currency in the discussion of data ethics. In this context, it's important because it's a warning against abstraction. Data abstracts people; data is an abstraction that becomes a proxy for you and your concerns—forgetting that data is people is dangerous in many ways, not in the least because these abstractions are often the basis for decisions, ranging from loans and mortgages to bail and prison sentences. Training data is inherently historical and reflects a history of discrimination and bias. That history is inevitably reflected when our automated abstraction engines do their job. Building systems that are fair requires transcending, or rejecting, the way data abstracts people. And it requires thinking forward about a future we would like to build—not just a future that’s a projection of our abstracted and unjust past.

I've written a couple of times about AI, art, and music. On one level, music theory is a massive set of very formal abstractions for understanding Western music, but great advances in music take place when a musician breaks all those abstractions. If an AI system can compose significant music, as opposed to "elevator music," it doesn't just have to learn the abstractions; it has to learn how to struggle against them and break them. You can train an AI on the body of music written before 2000, but you can't then expect it to come up with something new. It's significant that this is exactly the same problem we have with training an AI to produce results that are "fair."

The same is arguably true for literature. Is a poem just a cleverly arranged sequence of words, or is poetry important to us because of the human sensibility behind it? But the problem goes deeper. The notion that a poem is somehow about an author's feelings only dates back to the late 18th century. We don't hesitate to infer it in much earlier works, though it's not something those authors would have recognized. So we have to ask: what makes Chaucer a great storyteller, if most of his stories were already floating around in the literary tradition? What makes Shakespeare a great playwright, as most of his plays relied heavily on other sources? It's not the creation of the stories themselves, but something to do with the way they play against tradition, politics, and received ideas. Can that be represented as a set of abstractions? Maybe, but my instinct tells me that abstraction will give us little insight, though it may give us great poetry for our greeting cards.

I've talked with people recently who are doing amazing work using AI to attempt to understand animal cognition and language. There's no shortage of animals that demonstrate "intelligence": Alex the Parrot, Koko the gorilla, and so on. One problem with our search for animal intelligence is that we ask animals to be intelligent about things that humans are intelligent about. We ask them to be competent humans, but humans would almost certainly be terribly incompetent parrots or gorillas. We have no idea what a parrot thinks about the feel of air on its wings. We do know that they're capable of abstraction, but do our abstractions map to theirs? Probably not—and if not, is our drive to abstraction helping, or getting in the way?

What else could we imagine if our desire for abstraction didn't get in the way? The human mind’s ability to form abstractions is certainly one of our defining characteristics. For better or worse (and often for worse), we’re great at sorting things into categories. But we also do other things that aren’t modeled by our AI abstractions. We forget, and “forgetting” may be much more important to our mental abilities than we think; we form models with minimal data; we change our minds, sometimes freely, sometimes with great reluctance; we get bored; we find things “interesting” or “beautiful” for no reason other than that they please us. And we make mistakes—as does AI, though for different reasons. When we think about a future AI, should we be thinking about these capabilities?

If you go back to the grandparent of all AI metrics, Turing's "Imitation Game," and read his paper carefully, you find that Turing's computer, playing a game with a human interlocutor, doesn't always give the correct answer. A human talking to an oracle that is always right would have every reason to believe that the oracle wasn't human. Even at the beginning of computing and AI, there's a recognition that our abstractions (like "an artificial intelligence answers questions correctly") are heading us in the wrong direction.

Perhaps those errors are where the real value lies in our quest for a richer AI. It's fundamental that any trained system, whether you call it machine learning or artificial intelligence, will make errors. If you've trained a system so that it performs perfectly on training data, you have almost certainly overfitted, and it likely will perform very badly on real-world data. But if we don’t expect the system to be perfect on the training data, we clearly can’t expect it to perform better on real-world data. Let's ask a more provocative question: what if the erroneous results are more important than the correct ones? That wouldn't be the case if the AI is designed to detect disease in tomatoes. But if the AI is attempting to create interesting music, those might be precisely the results we need to look at. Not "here's another piano piece that sounds sort of like Mozart, Beethoven, and Brahms," but "here's a piece that breaks the rules and makes you rethink how music is structured." "Here's a story that's told in radical ways that Shakespeare could never have imagined."

I'm afraid I've made Chapman's compelling problem of evaluating progress worse. But this is the question we need to ask: what could we build if we stepped away from abstraction? As Hume suggests, Eve could have done so much more than aspire to a higher place on the hierarchy; she could have rejected the hierarchy altogether. If we tried to build our AIs along those lines, would it amaze us or scare us? Do we even know how to think about such software?

No one said it would be easy.

Continue reading Why we need to think differently about AI.

Categories: Technology

5 findings from O'Reilly's machine learning adoption survey companies should know

O'Reilly Radar - Tue, 2018/08/07 - 05:30

New survey results highlight the ways organizations are handling machine learning's move to the mainstream.

As machine learning has become more widely adopted by businesses, O’Reilly set out to survey our audience to learn more about how companies approach this work. Do companies with more experience deploying machine learning in production use methods that differ significantly from organizations that are just beginning? For companies that haven’t begun this journey, are there any best practices that might help?

Looking at mainstream adoption for machine learning now—especially in light of recent data privacy legislation such as General Data Protection Regulation (GDPR) in Europe and a similar political movement in California—we wanted to probe current trends, with these questions in particular:

  • How experienced are companies with machine learning adoption, in terms of number of years deploying models in production?
  • What has the impact been on culture and organization—for example, have job titles changed?
  • Who builds machine learning models: internal teams, external consultants, cloud APIs?
  • How are decisions and priorities set and by whom within the organization?
  • What methodologies apply for developing machine learning—for example, Agile?
  • What metrics are used to evaluate success?

Notable findings from the survey include the following:

  • Job titles specific to machine learning are already widely used at organizations with extensive experience in machine learning: data scientist (81%), machine learning engineer (39%), deep learning engineer (20%).
  • One in two (54%) respondents who belong to companies with extensive experience in machine learning check for fairness and bias. Overall, 40% of respondents indicated their organizations check for model fairness and bias. As tutorials and training materials become available, the number of companies capable of addressing fairness and bias should increase.
  • One in two (53%) respondents who belong to companies with extensive experience in machine learning check for privacy (43% across respondents from all companies). The EU’s GDPR mandates “privacy-by-design” (“inclusion of data protection from the onset of the designing of systems rather than an addition”), which means more companies will add privacy to their machine learning checklist. Fortunately, new regulations coincide with the rise of tools and methods for privacy-preserving analytics and machine learning.
  • One in two (51%) respondents use internal data science teams to build their machine learning models, whereas use of AutoML services from cloud providers is in low single digits, and this split grows even more pronounced among sophisticated teams. Companies with less-extensive experience tend to rely on external consultants.
  • Sophisticated teams tend to have data science leads set team priorities and determine key metrics for project success—responsibilities that would typically be performed by product managers in more traditional software engineering.

For a deep dive into these insights and more, download the free report The State of Machine Learning Adoption in the Enterprise with the full findings from our ML adoption survey.

Continue reading 5 findings from O'Reilly's machine learning adoption survey companies should know.

Categories: Technology

Four short links: 7 August 2018

O'Reilly Radar - Tue, 2018/08/07 - 04:05

Azure Kubernetes, Systems Neuroscience, Facebook's TLS, and Speech Benchmark

  1. Horrors of using Azure Kubernetes Service in Production -- a cautionary tale.
  2. Systems Neuroscience is About to Get Bonkers -- Two advances are making the impending data surge an important issue to address now. The first is the advent of widely available, low-cost recording technologies that are easy to use, such as the Neuropixels probes. [...] The second is the maturation of deep learning, a catchphrase for a collection of very powerful artificial neural network concepts, along with the software and hardware that power them.
  3. Fizz -- Facebook's TLS 1.3 implementation, open-sourced. (via Facebook blog post)
  4. stt-benchmark -- a minimalist and extensible framework for benchmarking different speech-to-text engines.

Continue reading Four short links: 7 August 2018.

Categories: Technology

Case studies in data ethics

O'Reilly Radar - Tue, 2018/08/07 - 04:00

These studies provide a foundation for discussing ethical issues so we can better integrate data ethics in real life.

To help us think seriously about data ethics, we need case studies that we can discuss, argue about, and come to terms with as we engage with the real world. Good case studies give us the opportunity to think through problems before facing them in real life. And case studies show us that ethical problems aren't simple. They are multi-faceted, and frequently there's no single right answer. And they help us to recognize there are few situations that don't raise ethical questions.

Princeton's Center for Information Technology Policy and Center for Human Values have created four anonymized case studies to promote the discussion of ethics. The first of these studies, Automated Healthcare App, discusses a smartphone app designed to help adult onset diabetes patients. It raises issues like paternalism, consent, and even language choices. Is it OK to “nudge” patients toward more healthy behaviors? What about automatically moderating the users’ discussion groups to emphasize scientifically accurate information? And how do you deal with minorities who don’t respond to treatment as well? Could the problem be the language itself that is used to discuss treatment?

The next case study, Dynamic Sound Identification, covers an application that can identify voices, raising issues about privacy, language, and even gender. How far should developers go in identifying potential harm that can be caused by an application? What are acceptable error rates for an application that can potentially do harm? How can a voice application handle people with different accents or dialects? And what responsibility do developers have when a small experimental tool is bought by a large corporation that wants to commercialize it?

The Optimizing Schools case study deals with the problem of finding at-risk children in school systems. Privacy and language are again an issue; it also raises the issue of how decisions to use data are made. Who makes those decisions, and who needs to be informed about them? What are the consequences when people find out how their data has been used? And how do you interpret the results of an experiment? Under what conditions can you say that a data experiment has really yielded improved educational results?

The final case study, Law Enforcement Chatbots, raises issues about the tradeoff between liberty and security, entrapment, openness and accountability, and compliance with international law.

None of these issues are simple, and there are few (if any) "right answers." For example, it’s easy to react against perceived paternalism in a medical application, but the purpose of such an application is to encourage patients to comply with their treatment program. It’s easy to object to monitoring students in a public school, but students are minors, and schools by nature handle a lot of private personal data. Where is the boundary between what is, and isn’t, acceptable? What's important isn’t getting to the correct answer on any issue, but to make sure the issue is discussed and understood, and that we know what tradeoffs we are making. What is important is that we get practice in discussing ethical issues and put that practice to work in our jobs. That’s what these case studies give us.

Continue reading Case studies in data ethics.

Categories: Technology

Four short links: 6 August 2018

O'Reilly Radar - Mon, 2018/08/06 - 04:05

GPU Notebooks, Reproducible Experiments, History of Fears, and Debloating Code

  1. Kernels -- Kaggle hosting Jupyter Notebooks with GPUs.
  2. Speedrun -- A no-strings-attached toolkit to help you deploy and manage your machine learning experiments. The idea is to equip you with the tools you need to have well-documented and reproducible experiments going, but without getting in your way.
  3. Ancient Dreams of Intelligent Machines (Nature) -- the extraordinary history of cultural responses to automata.
  4. Trimmer -- an application specialization tool that leverages user-provided configuration data to specialize an application to its deployment context. The specialization process attempts to eliminate the application functionality that is unused in the user-defined context. Our evaluation demonstrates Trimmer can effectively reduce code bloat. For 13 applications spanning various domains, we observe a mean binary size reduction of 21% and a maximum reduction of 75%. Description of how it works, but no source, alas.

Continue reading Four short links: 6 August 2018.

Categories: Technology

5 things you should be monitoring

O'Reilly Radar - Mon, 2018/08/06 - 03:00

Achieve high-impact systems monitoring by focusing on latency, errors, throughput, utilization, and blackbox monitoring.

So you've deployed your brand new web application (now with blockchain!), and your users love it. Traffic is increasing, and you're getting great press. Then one morning you wake up to find that the site has been slow all night, and your users are complaining. How do you find the problem?

Whether you're a developer building websites or internal applications, or an administrator building the infrastructure to back them, your job doesn't stop once they're up and running. Machine failure, releases containing bugs, and growth in usage can all lead to problems that need to be dealt with. To detect them, you need monitoring.

Continue reading 5 things you should be monitoring.

Categories: Technology

Topics for the August 9th meeting

PLUG - Sat, 2018/08/04 - 09:33
We'll be having 2 presentations this month from Ed Nicholson and der.hans.

Ed Nicholson - Technical Debts and Digital Assets

Exploring the analogy of debt and real world financial obligations, as
a context for a software's or system's life cycle.
Debt and Software are both fundamentally types of Intangible Assets.
This talk is an exploration of how the much older asset class, Debt,
may be usefully applied in understanding the relative newcomers,
Software and Systems.

About Ed:
I support Free Software and how people, organizations and populations use, create and experience information. I enjoy The Prescott National Forest as my "office" and The Valley of The Sun as a home away from home. Personal systems run either Fedora, Project Atomic, Android or OpenBSD.

der.hans - Introduction to /etc/alterntives

/etc/alternatives allows choosing among many programs to provide a certain functionality.
For instance, editor can point the locally preferred text editor. alternatives will also make sure man-pages also match.
Attendees will learn:
  • What is /etc/alternatives
  • Why is /etc/alternatives
  • How to configure /etc/alternatives
  • How to use helper scripts
About der.hans:
der.hans is a Free Software community veteran, presenter and author. He is the founder of the Free Software Stammtisch, BoF organizer for the Southern California Linux Expo (SCaLE) and chairman of the Phoenix Linux User Group (PLUG).

As a technology and entrepreneurial veteran, roles have included director of engineering, engineering manager, IS manager, system administrator, community college instructor, developer and DBA.

He presents regularly at large community-led conferences (SCaLE, SeaGL, LibrePlanet, LFNW, Tübix) and many local groups.

10 practical talks you won’t want to miss at the O’Reilly Velocity Conference in New York 2018

O'Reilly Radar - Fri, 2018/08/03 - 03:00

Get advice and insight from speakers who have tackled the challenges you face.

We’re excited by the lineup at the O’Reilly Velocity Conference in New York this year. Our core theme is building better systems: faster, more secure, and more resilient. To this end, we’ve gathered an amazing collection of speakers, talking about the technologies and challenges facing systems engineers today. Our focus has been on finding speakers with real world experience, both successes and failures, who can help you solve (or avoid!) problems in your own environment.

It’s hard to highlight all the talks we love, but we’ve selected 10 talks (aside from our keynotes!) that are unmissable: covering the gamut from serverless, security, and Kubernetes, to monitoring, and leadership.

Securing serverless: By breaking in — Guy Podjarny, Snyk

Serverless is crossing the chasm and we’re starting to see large-scale real-world implementations of serverless for mission-critical applications. Serverless removes some security threats, but new and different threats are emerging. Guy Podjarny will take you through a live attack of a serverless application and show you how you can protect your serverless environments.

Deprecating simplicity — Casey Rosenthal, Backplane

Casey Rosenthal, who co-wrote the book on chaos engineering, is going to take you through the architecture, system thinking, and engineering priorities you need to consider when building complex distributed systems. He’ll explain how chaos engineering allows you to navigate complexity so you can increase feature velocity without sacrificing resiliency, security, or your sanity. 

Rebuilding the airplane in flight … safely — Shannon Weyrick and James Royalty, NS1

NS1 runs one of the largest and most complex DNS infrastructures in the world. Shannon Weyrick and James Royalty will take you through their journey rewriting the DNS server powering their infrastructure, all while avoiding customer impact. It’s a fascinating story of a complex engineering implementation.

Switching horses midstream: The challenges of migrating 150+ microservices to Kubernetes — Sarah Wells, Financial Times

The Financial Times was an early adopter of microservices and containers—the platform it built pre-dated production-ready Kubernetes. When the FT decided Kubernetes was ready for the mainstream, it faced the challenge of migrating more than 150 services to a new platform, all without downtime. Sarah Wells will talk about why the Financial Times migrated and how it managed to do so with minimum interruption to service.

Communicating and managing change — Rocio Delgado, Slack

Every organization changes. Some changes are handled well, others can cause immense harm to a company and its culture, even minor changes that are simply communicated or implemented poorly. Rocio Delgado will show you how to make healthy change in your organization and ensure good communication, trust, and buy-in.

Building successful SRE in large enterprises — Liz Fong-Jones and Dave Rensin, Google

Liz Fong-Jones and Dave Rensin will take their experiences at the home of SRE, Google, and share them with a wider community. Want to understand what building an SRE team looks like in an enterprise world? This is the roadmap for you. Oh, and there’s also this if you want to get a head start.

The container operator's manual — Alice Goldfuss, GitHub

Running containers in development is one thing. Running containers at scale for mission-critical services is another. Alice Goldfuss will take you through the real world operational life of GitHub’s large-scale container environment.

How to break up with your vendor — Amy Nguyen, Stripe

We’ve all experienced vendor lock-in. Sometimes you don’t mind being locked in with your vendor and sometimes you do. Like when the relationship goes bad and turns unresponsive, unhelpful, or takes the product in directions that aren’t good for your use case. Breaking up with a vendor is hard, involving potential rework and disruption. Amy Nguyen will take you through the steps of making a vendor transition work with minimum fuss.

Modern security best practices for microservices and distributed systems — Seth Vargo, Google

The days of perimeter-only-centric defense are truly over. Applications require true defense in depth, with security input needed for architecture, design, development, deployment, and operations. Seth Vargo will focus on how to make sure your microservices and distributed systems get the security implementation they need through collaboration and code.

From silos to a single pane of glass at USA TODAY NETWORK — Bridget Lane and Kris Vincent, Gannett & USA TODAY

Gannett & USA TODAY had a complex and siloed environment. Bridget Lane and Kris Vincent will take you through their journey breaking down those silos, building cross-functional teams, and improving the tools and systems managing their infrastructure. It’s a fascinating look at a complex and successful DevOps migration.

We hope this has given you a taste for the larger Velocity New York program. We found it really hard to narrow our highlights to just 10 talks and we think you’ll find it equally hard (in a good way) to decide what to see at the conference. We hope to see you all in New York in October!

Continue reading 10 practical talks you won’t want to miss at the O’Reilly Velocity Conference in New York 2018.

Categories: Technology

Four short links: 3 August 2018

O'Reilly Radar - Fri, 2018/08/03 - 01:00

Text to Faces, Cartoon Camera, Change Tracking, and Quantum Mechanics Toys

  1. text2face -- turns a textual description of a face into a photo. Creates rather ghostly images now, but you can easily see this as the start of an automated identikit.
  2. Draw This -- an instant camera that draws cartoons. A whimsical deep learning application.
  3. Orbit -- a composable framework for orchestrating change processing, tracking, and synchronization across multiple data sources.
  4. These Quantum Mechanics Toys Didn't Catch On (IEEE) -- But the quantum toys proved a pedagogical nonstarter. Papaliolios never published an instruction manual for them or a paper describing their potential value in the classroom. When he lent them to colleagues or students, they were met with confusion or indifference. I'm imagining a CS Unplugged for quantum mechanics ...

Continue reading Four short links: 3 August 2018.

Categories: Technology

How privacy-preserving techniques can lead to more robust machine learning models

O'Reilly Radar - Thu, 2018/08/02 - 04:00

The O’Reilly Data Show Podcast: Chang Liu on operations research, and the interplay between differential privacy and machine learning.

In this episode of the Data Show, I spoke with Chang Liu, applied research scientist at Georgian Partners. In a previous post, I highlighted early tools for privacy-preserving analytics, both for improving decision-making (business intelligence and analytics) and for enabling automation (machine learning). One of the tools I mentioned is an open source project for SQL-based analysis that adheres to state-of-the-art differential privacy (a formal guarantee that provides robust privacy assurances).  Since business intelligence typically relies on SQL databases, this open source project is something many companies can already benefit from today.

What about machine learning? While I didn’t have space to point this out in my previous post, differential privacy has been an area of interest to many machine learning researchers. Most practicing data scientists aren’t aware of the research results, and popular data science tools haven’t incorporated differential privacy in meaningful ways (if at all). But things will change over the next months. For example, Liu wants to make  ideas from differential privacy accessible to industrial data scientists, and she is part of a team building tools to make this happen.

Continue reading How privacy-preserving techniques can lead to more robust machine learning models.

Categories: Technology

4 steps for creating design principles

O'Reilly Radar - Thu, 2018/08/02 - 04:00

Principles help you align your team quickly around a shared set of product guidelines.

Continue reading 4 steps for creating design principles.

Categories: Technology

4 personal and tool-based methods for creating strong feedback loops

O'Reilly Radar - Thu, 2018/08/02 - 03:00

Practical examples of how to integrate personal and tool-based feedback into your code review process.

In one of my previous posts, I explained how strong feedback loops make strong software teams and how the most effective feedback loops are a mixture of daily best practices, automation, tools, and human intervention. Good quality control combines tool-based measurement with manual review and discussion.

This follow-up post offers examples of specific practices that integrate personal and tool-based feedback, which will help you bolster your code and architectural quality. These four practices might already be part of your day-to-day operations, but you need to make sure you’re getting the most out of them in terms of increasing your code and product quality.

1. Pull requests and code reviews

Pull requests let other team members know if code is ready to be merged and deployed without requiring them to check statuses continuously. It will also allow team members to do code reviews before deployment and provide feedback. Various research—including a survey conducted by SIG and O’Reilly of more than 1,400 developers, as well as other studies—shows that peer-led, manual code reviews are the most effective method to address code quality during the development process. This research also shows that tools usage is the second most effective method. It stands to reason, then, that a combination of interpersonal and tool-based feedback yields the highest quality code during review.

2. Static analysis tools

Objective, tool-based assessment provides immediate, actionable feedback on the quality of your code so you know what to improve and where to start. The aforementioned research shows that the most valued capabilities of code quality tools are finding security vulnerabilities, identifying violations of software design guidelines, and providing metrics on test coverage. The most effective tools for your team may vary, but static analysis tools are integral to a rigorous code review process.

3. Autonomous teams

Teams having to wait for each other and hand over phases will slow down your development process. Autonomous teams can establish their own shared goals, language, and boundaries—a “shared sense of reality,” so to speak. Your technical architecture needs to be mapped to these teams, because different teams need to work in separate code bases. This autonomy will decrease dependencies and increase your teams’ productivity.

4. Retrospective and daily scrum

Use these occasions to reflect on goals, progress, dependencies, and alignment. Doing so will help teams maintain that shared sense of reality mentioned above. Make sure product owners and users are part of this feedback loop, too.

Daily scrums are also perfect opportunities to agree on measurable definitions of “done” for each team, which will allow you to keep track of progress and maintain quality.

A shared reality

When teams have common standards, metrics, and techniques, they are more productive. Moreover, regularly validating and reflecting on this shared reality through feedback loops is crucial for product quality.

This list is obviously not exhaustive, but it helps to illustrate how tool-based and personal feedback reinforce each other and enable efficiency. With an effective, rigorous code review process in place, development teams can spend their time actually addressing any issues the review process uncovers, allowing them to focus on the right priorities and be as productive as possible.

This post is a collaboration between O'Reilly and SIG. See our statement of editorial independence.

Continue reading 4 personal and tool-based methods for creating strong feedback loops.

Categories: Technology

Four short links: 2 August 2018

O'Reilly Radar - Thu, 2018/08/02 - 01:00

Problem Recognition, Evolving Floorplans, Google in China, and The Bullshit Web

  1. Clues and Signals -- a product manager's pattern library for when things are going to go wrong. So on-point it burns. Samples: Might as well [do some extra thing] while we [do the original thing]. It's too early to [some interaction with users/customers]. We need some quick wins because [normal wins take too long].
  2. Evolving Floorplans -- The rooms and expected flow of people are given to a genetic algorithm which attempts to optimize the layout to minimize walking time, the use of hallways, etc.
  3. Google Returning to China, with Censorship (The Intercept) -- “I’m against large companies and governments collaborating in the oppression of their people, and feel like transparency around what’s being done is in the public interest,” the source said, adding that they feared “what is done in China will become a template for many other nations.” Nicely said. “Organize the world’s information and make it universally accessible and useful," with a few key caveats.
  4. The Bullshit Web -- An honest web is one in which the overwhelming majority of the code and assets downloaded to a user’s computer are used in a page’s visual presentation, with nearly all the remainder used to define the semantic structure and associated metadata on the page. Bullshit — in the form of CPU-sucking surveillance, unnecessarily-interruptive elements, and behaviours that nobody responsible for a website would themselves find appealing as a visitor — is unwelcome and intolerable.

Continue reading Four short links: 2 August 2018.

Categories: Technology

How to design bot conversations

O'Reilly Radar - Wed, 2018/08/01 - 04:00

Tools to fine-tune the wording, formatting, and other aspects of conversation to render rich interactions.

I love taking an idea to a prototype, and then to a product that millions of people use.


There are a lot of ways to design a conversation. One option is to just expand on the Wizard of Oz technique demonstrated in Chapter 15, and mimic the scripts by impersonating the bot. While this is a very easy and quick method to get your product in front of your users and other stakeholders, it provides low fidelity when it comes to rendering rich interactions. This is because the chat platforms limit the types of rich controls available to humans. Users can post simple images, GIFs, and even videos, but they cannot display buttons, for example.

When it comes to software solutions, there are also a lot of design tools that provide you with different levels of fidelity and ease of use. There are many good options, and you should pick the ones that suit you. I have chosen two tools as examples, one for designing bots for Facebook Messenger and the other for Slack.

In the next few sections, we will go over the scripts we created in the previous chapter and use these design tools to visualize these scripts. For each script, we will try to fine-tune the wording, formatting, and other aspects of the conversation. This is an iterative process that will demonstrate design in real life.

Designing VacationBot for Facebook Messenger with Botsociety

Let’s start with a tool called Botsociety ( Botsociety is a super-easy and quite full-featured design tool for bots. At the time of writing Botsociety supports only Facebook Messenger, but the team have told me they are planning to launch support for other platforms very soon.

After registering, we will choose a name for our bot, select a platform, and start sketching (Figure 1-1).

Figure 1-1. Creating a mockup with Botsociety

Next, we’ll go into the main designing area. As you can see, the tool uses a super-simple “Bot Says”/“User Says” paradigm (Figure 1-2).

Figure 1-2. Drafting the conversation script (“Bot Says”/“User Says”)

Clicking on the “Bot Says” button provides you with a choice of the common types of output bots on the Facebook Messenger platform can provide (Figure 1-3).

Figure 1-3. Output types available to bots on Messenger

Similarly, clicking on “User Says” offers you a choice of the types of input available to users on Facebook Messenger (Figure 1-4).

Figure 1-4. Input types available to users on Messenger

From a quick browse of the elements on both sides of the conversation, it looks like we have everything we need to start designing our scripts. So let’s give it a try! I have taken the onboarding script outlined in the previous chapter for our VacationBot and entered the first part of it into the tool. Immediately, I notice an issue with the original script—it’s so long that it runs below the fold (meaning you need to scroll read it all). This makes for a bad user experience, as it is hard to understand at a glance what the bot is and what it wants from the user (Figure 1-5).

Figure 1-5. VacationBot’s onboarding script—the user needs to scroll to read all of the text

I have a bad feeling about this: if users do not see the entire value proposition up front, they might just back out of the conversation. Let’s make it shorter (Figure 1-6).

Figure 1-6. A shortened version of the onboarding script

I made the bot’s value proposition a little more concise (which is a good thing on its own), and now everything fits well within the window without scrolling. We will, of course, need to test it on several devices, both mobile and desktop, but this is a good start.

Note Key takeaway

The difference between a good experience and a poorly executed one can be in the small details, such as how long the text is and whether the user has to scroll to read all of it.

Some designers will prefer to do all of their scripting in tools like this, for the benefit of seeing immediately how the script looks in real life—if you feel more comfortable doing so, please do. I prefer to start with written scripts, as it enables me to really think about the flow in the context of multiple scripts and use cases. It is also easier to cut, paste, and share initial scripts written as text rather than as a GIF, which is the output of this tool.

Now we can continue with the script and see if the rich interaction we envisioned works well. Botsociety provides you with the ability to render more than plain text: it also enables you to render rich controls like carousels. Near the end of our onboarding script, we have a section where we demonstrate the value of the bot with a carousel of activities at the vacation destination. We might need to wait for user feedback, but I think the outcome is quite nice (Figure 1-7).

Figure 1-7. The carousel of attractions

Finally, we can explore the call to action at the end of our onboarding script (Figure 1-8).

Figure 1-8. The call to action

As you can see, not all the buttons are visible, and the user needs to scroll to see all the options. This could actually be a blessing in disguise, as the last option—the one the users need to scroll to pick—is the one that we do not want them to click on (the option that declines the offer). The positioning of the UX elements, both on the screen and off, subtly encourages the user to pick one of the “right” choices and subscribe to the bot’s feed. Of course, users are still able to decline by clicking on “No thanks” or just not clicking on anything.

This brings to light another consideration we will need to take into account—if the user does not click on any of the buttons, we will have to treat it as a “No thanks” after a certain amount of time, and continue on to provide the user with a way to ask for recommendations manually at any time.

By now we have tested all of the rich controls we have planned for our VacationBot, for this use case. Let’s finish up the onboarding design, and pick the “No thanks” Quick Reply (Figure 1-9).

Figure 1-9. Handling the “No thanks” reply

It looks like we can do a little better—we are missing the opportunity to let the user schedule reminders at a later stage. Let’s add that now (Figure 1-10).

Figure 1-10. Adding a Quick Reply to schedule notifications

This looks much better now, and the user has another option to subscribe to the service. Note that in the previous design, after users have declined the offer to subscribe, they do not have a way to back out of that decision. It is always recommended to give the user the option to reconsider and do the right thing.

Now we will implement VacationBot’s main flow (Figure 1-11).

Figure 1-11. Implementing the main flow

I think it looks pretty good. Remember, we will have a “Schedule notifications” option in case the user has not done so. Here, we assume the user has subscribed to receive notifications; the top text saying “Hello again!” would not be visible in cases where notifications are turned off.

Now let’s design the help script. We will add Quick Replies at the end of the help text to prevent a dead end (Figure 1-12).

Figure 1-12. The help script

This is better than the original script because it is more consistent and always provides the user with options for what to do next.

Now let’s implement the feedback script of our VacationBot (Figure 1-13).

Figure 1-13. The feedback script

I have added the consistent ending that gives the user a hint about what to do next. I really like the outline of the Great...Terrible Quick Replies. One issue is that the bot does not acknowledge the user’s feedback rating; this might be fine, but we might want to test if users find that awkward or not.

Finally, we will design a generic error script. From the design up to now, I already know to add the standard “Recommendations” Quick Reply at the end of the conversation (Figure 1-14).

Figure 1-14. The generic error handling script

The last thing I’ve noticed is that the logo looks really bad—the text is not visible, and it looks small. We will fix this for both bots later in this chapter.

As you can see, we have learned a lot from just doing a simple visualization of our scripting. We have noticed places where we can improve user engagement by adding Quick Replies and avoiding dead ends. We have also seen how the rich controls look, in an environment that is close to real life, and have modified our text to improve the layout of our conversations.

Designing PTOBot for Slack with Walkie

In order to design our PTOBot on Slack, I am going to use a design tool called Walkie ( Walkie is a flexible and a feature-rich tool that lets you script multiple flows. It all starts with setting up your bot and user (Figure 1-15).

Figure 1-15. Getting started with Walkie

After saving the settings, we go into a Slack-like user experience (Figure 1-16).

Figure 1-16. Walkie’s Slack-like UI

On the left there is a list of bots that we’ve created (in this case, PTOBot), then there is a list of flows which are distinct scripts (PTO approval, for example). The main area to the right is the design section, with a place to enter user and bot inputs. Clicking on the “User” button toggles between bot and user. There is also a control at the bottom right to create rich interactions through message attachments. Clicking on it opens up a fully configurable message attachment, including buttons (technically called attachment actions; see Figure 1-17).

Figure 1-17. Adding a message attachment

The tool does a good job of supporting multiple bots, but does not support multiple users (user personas) interacting with a bot. So, I will create a few bot configurations to work around that limitation.

Let’s start with the onboarding script. The first conversation is with the user who installed the bot (Figure 1-18).

As you might recall, the onboarding script shows a GIF of the PTO request process, in order to demo the bot’s usage. Showing this in a printed book is a challenge on its own, so I have cheated a little and used a screenshot of the PTO request process.

Figure 1-18. The onboarding conversation with the bot installer

The script looks okay, but it suffers from the opposite problem our VacationBot had in its onboarding script—the script here seems too dry and too short, and I am not sure it is clear and actionable enough.

Let’s try to fix that (Figure 1-19).

Figure 1-19. Fleshing out the installer onboarding script

I added a few emojis, made the text a little more descriptive, and added a button at the bottom of the script that lets the installer invite the bot to the right channels with a single click. Because the Slack API lets us to add a bot to a channel programmatically, we can use this button to shortcut the need for the user to go into the relevant channels and invite the bot manually. We will use the API to add the bot automatically, while still giving the control to the installer, by only adding the bot after the button has been clicked.

I also started to add a color convention: blue will be informative (like the blue color next to the demo GIF) and green actionable, for actions we want the user to perform.

The entire onboarding fits on a single page, without scrolling, on a web interface. You should not be too worried about this in a work context, but it is still best practice to keep the initial conversation above the fold.

Let’s continue to the next step, the team onboarding script, shown after the bot is invited to a channel by the installer (Figure 1-20).

Figure 1-20. The team onboarding script

The text is very similar to the installer script, but you will notice I have added a little text decoration at the end of the team onboarding text. I surrounded the slash command with backticks (``) to render the text as a code block. This hints to the user that the slash command is like a short command line that they can use, and that they should pay attention to the parameters the command accepts (in the same way one does when running a script on the command line).

Now, let’s move on to the main flow. To remind you, the main functionality of our PTOBot is as follows:

  1. Employee requests PTO in a direct message with the bot (or a slash command).

  2. Manager gets a notification and approves/rejects the request.

  3. Employee gets notification of approval/rejection.

  4. Team gets notification of PTO.

This is by no means a simple “Hello World”–style process. I did not want to avoid complexity, but wanted to demonstrate the flexibility and unique attributes possible in bots for a work environment. We will design each step in this process, learning and improve each step along the way.

We’ll design each of these steps in a separate Walkie flow, starting with the PTO request (Figure 1-21).

Figure 1-21. The PTO request script

You will notice that I have used some lightweight formatting by making the dates and the description captured stand out in bold (surrounding them with *s) and kept the green color coding for actions we would like the user to take.

As you can see, the conversation is long with a few places for potential errors. This is where our Slack command comes into play. Let’s see the same conversation compacted to a couple of lines (Figure 1-22).

Figure 1-22. Designing the slash command interaction

In a single line the user has provided all the necessary information to the bot, initiating a PTO request without the need for a lengthy conversation. Slash commands are great when you have a small and structured set of entities (variables) your bot needs to extract, and a savvy set of users who can remember how to use the commands.

Now, let us continue to the manager approval step (Figure 1-23).

Figure 1-23. The manager PTO approval script

This is OK, but it could be better. The name of the game here is get things done as fast as possible. This mean rendering the information in the easiest possible way to digest. We made the important parts bold, but I think the way the message is currently structured forces the user to read through it in order to get the necessary information. Let’s see if we can enter the details in Slack’s structured template (called a message attachment) and make it easier to digest and act upon. Figure 1-24 shows the result.

Figure 1-24. Rendering the request details in a message attachment

I think this might be an easier way for the manager to pull out the relevant data. Of course, we will have to test it with actual users, as this is only an assumption.

Now let us finish up the approval step (Figure 1-25).

Figure 1-25. Continuing the approval script

Now that we have designed the script, you might notice a few shortcomings with this design. The buttons are still there, and there is a chance the user will click on them by mistake. There is also a good chance that this design will be messy in a real-life scenario, when multiple requests might come in concurrently. It will be hard to manage the requests and keep track of which have been approved and which were rejected. Let’s try another approach (Figure 1-26).

Figure 1-26. Fine-tuning the approval script

In this design I replaced the buttons, once the user has clicked “Approve,” with an approval confirmation. I think this is a better way to implement the process. Replacing the buttons ensures the user does not press one of them again by mistake. It also removes some of the cognitive load, if a lot of messages like this one appear in a conversation, and gives the user the feeling of accomplishment that users love in todo lists.

In the Walkie tool itself, I have forked the approval flow into “request approved” and “request rejected.” Figure 1-27 shows what the “rejected” flow looks like.

Figure 1-27. The “request rejected” flow

Now it is easy to see at a glance which requests have been accepted or rejected, and the user does not need to read through a text conversation to see which requests have been handled. In more advanced versions we might want to add a reason for rejection, but let’s keep it simple for now.

Next, in the employee notification flow, we will implement what we’ve learned about message formatting and button replacement (Figure 1-28).

Figure 1-28. The employee notification script

Notice that the bot is actually rendering two message attachments—one is informational, color coded in blue, and the other is actionable, color coded in green. Clicking on “Notify Team” will follow the same practice of replacing the buttons with a confirmation that the notification has been sent (Figure 1-29).

Figure 1-29. Replacing the buttons with a confirmation

Note that we also changed the color coding of the second attachment to blue as it moved from actionable to informational. I chose to use this color schema as an example of how color coding can help with mental load reduction. We will have to test if this resonates with our users later on.

Let’s finish this step by suggesting that the user install VacationBot (Figure 1-30).

Figure 1-30. Recommending VacationBot

This is a unique pattern of one bot recommending another bot to the user, and on a different platform. Clicking on the link will take the user straight to an onboarding conversation with VacationBot on Facebook Messenger.

Meanwhile, let’s finish the main flow by designing the team PTO notification (Figure 1-31).

Figure 1-31. The team PTO notification

A PTO process like this is traditionally done manually, using paper forms, spreadsheets, or web tracking tools; it can be messy and require a lot of time. Our assumption is that users will find this process easier, more intuitive, and more productive.

The last thing to take care of is the logo. As mentioned earlier, particularly in our VacationBot, it looks small and indistinct. We also want the logo to be consistent in both bots. Moving forward, I will use the simple logo shown in Figure 1-32.

Figure 1-32. The new logo for VacationBot and PTOBotUser Testing

Now that we have designed the main flow of both bots, it is time to put them in front of actual users. First, we need to decide how we want to test our design.

There are a few options:

  1. Show users a video or a step-by-step replay of the conversation and get their inputs.

  2. Create a mock (fake) bot and let users play with it.

  3. Create a working alpha and let users work with it.

Both Botsociety and Walkie support replaying the conversation either as a movie or a step-by-step walkthrough. Showing potential clients/users these videos can get you very valuable feedback. You will not be able to see users perform tasks themselves, which might be the most important indication of good design, but you will get feedback fast and with little development cost.

As for mocking a bot, Walkie goes a step further and lets you export the conversations into a JSON-format file. An engineer can plug this file into a script that mimics an actual bot. A mock bot is a great tool for testing interactions, and it doesn’t really matter that the bot is not connected to the real backend system. In our case, we don’t care that the bot is not connected to a real PTO system, or that the data is fake. Getting a user to go through the request process and a manager to go through the approval process in real life will surely teach us a lot.

Note Key takeaway

A mock bot is a great tool for testing interactions. It doesn’t really matter that the bot is not connected to the real backend system.

Alternatively, if you are confident with your design, you might even start building the actual bot and get feedback from live alpha users. This is useful because it is the shortest path to production, if you get it right. Users test the real bot, with real data, and you get live and super-accurate feedback. This option will work well if you have clients who are willing and able to be your alpha users and use your software in real life.

Laura Klein has written a great “Step-by-Step Usability Testing Guide” ( that outlines the steps in usability testing. Assuming that we will create a mock that users can play with, let’s discuss the usability testing steps for our PTOBot.

Before You Start—Prototyping a Mockup Bot

First let’s create a mockup of our PTOBot. In order to prototype simple processes we will use a tool I developed called ProtoBot, which you can install freely by searching for ProtoBot in the Slack app directory (

ProtoBot does not require coding skills, and it is really easy to create mockups of bots with it. You install ProtoBot in a testing team of your choice, and start a conversation with it. ProtoBot provides a detailed description of how to use it, but we’ll go through a short example here.

ProtoBot can mimic multiple bots—that is why I initially called it “Dr Jekyl.” These bots are called personas, and ProtoBot can assume a persona with the following steps.

In a direct message with ProbotBot:

  1. Type /new-persona PTOBot to start a new bot persona script.

  2. Type /set-persona-name PTOBot to set the name the bot will use in this script.

  3. Type /set-persona-icon-url [URL] to set the icon the bot will use in this script.

  4. Say hello to your new bot (Figure 1-33).

Figure 1-33. ProtoBot learning to be PTOBot

The way to teach ProtoBot a new script is simple: you just start talking to the bot and follow its instructions (Figure 1-34).

Figure 1-34. ProtoBot tells you how to teach it a reply

/learn is a slash command you can use to teach ProtoBot what to say when the user says something. It follows this pattern:

  • /learn [user says]

  • [bot says]

Note the newline between what the user says and what the bot says (use Shift+Enter to create this newline in Slack). Let’s teach PTOBot what the script replies when the user says “I want to take a PTO” (Figure 1-35).

Figure 1-35. Teaching the bot a new script

Now, after we have trained ProtoBot with this step in the script, let’s run the same script again (Figure 1-36).

Figure 1-36. The bot has learned the correct reply

Yes! ProtoBot is starting to learn how to mimic the PTOBot persona.

In this way, using the /learn slash command, you can teach ProtoBot the entire script. It is important to note that ProtoBot does very little pattern matching, when it comes to user inputs—it is case insensitive and ignores characters like question marks or periods, but you will need to teach it the various permutations of possible user inputs for a given script. In this example, for instance, you can also teach the bot to handle inputs like “I want PTO” or “Start PTO request” to start the PTO request script.

ProtoBot has more advanced functionality, such as the ability to import JSON files from tools such as Walkie and support for multiple concurrent personas, but you can read all about it by just saying “help” to the bot in a direct message at any time. ProtoBot is a good tool for basic mockups, but it does not support complex scripts like contextual help. If you want to implement these, you might want to code your mockup or find a tool that supports these advanced features.

Once we have taught ProtoBot a section of the script, we can start thinking about the next steps in our usability testing.

Planning the Test

To test PTOBot we would like to find existing PTO-IT users—preferably friendly ones and early adopters—and invite them to PTO-IT’s office to do this experiment. We will also invite a few non-clients to see if VacationBot can be used as a standalone product.

We would like to invite users who are already using Slack, because we do not want to have to teach them how to direct message or how a channel works during the usability testing.

Creating Tasks and Discussion Guides

We will focus on two tasks: PTO requests and PTO approvals. We will create an overview of our PTOBot and collect responses to the following background questions from our participants:

  1. Managers:

    1. How many people do you manage?

    2. How many PTO requests do you handle in a month?

    3. What system or tool do you use for this today?

    4. Do you use bots for your day-to-day tasks?

    5. Do you use bots/Slack mainly on the web or on mobile?

  2. Employees:

    1. How many PTO requests do you make in a year?

    2. Who approves your PTO requests?

    3. What system or tool do you use for this today?

    4. Do you use bots for your day-to-day tasks?

    5. Do you use bots/Slack mainly on the web or on mobile?

    6. How do you let your team know you are taking PTO?

We will also create a task for each participant:


You are a busy exec with little time for paperwork. Show me how you use PTOBot to quickly get through your PTO approval tasks and get back to what’s important.


You want to take a vacation to Cancun for 10 days starting 04/21/17. Use PTOBot to get your paperwork in order.

Recruiting Participants

We will recruit four managers and four employees. Half of them should be current PTO-IT users and half of them will be new potential users who have not worked with PTO-IT in the past.

Setting Up the Environment

We will set up a clean computer with Slack installed and a test team setup. We will add ProtoBot to that team and teach it the PTO request script and the PTO approval script. Note that if it is easier for you to mock up the bot using any other tool, that’s fine too!

You can also set up the environment by opening Slack and showing the onboarding message from the bot in the HR channel, as that is the most likely place that users will come to learn how to use the bot.

Moderating the Sessions

Make sure to explain that this is just a prototype, and might not work perfectly or look final. Let each user try to perform the assigned task, and possibly fail, without interfering. Make sure to train ProtoBot well beforehand so that it can handle many permutations of user inputs. Also make sure to support the help script so that the user can get help if needed.

Make sure the users are comfortable and know it is OK to fail in these tasks. Ask them to speak their thoughts out loud while performing the task, and follow up with questions—for example, when a user makes a statement like “That was easy,” ask “What was easy about it?” Keep the same pattern for all types of comments. Take notes on all actions and comments made by the users.

Analyzing the Data

Mark each user test with “Task Completed,” “Task Completed with Difficulty,” or “Task Failed.” Correlate your notes on the comments and find patterns that signal issues. Here are some examples:

  1. Two employees had a hard time entering the start date of their PTO in a way the bot can understand.

  2. Three managers thought the PTO approval process was “very easy.”

  3. One employee started a PTO request in the channel rather than in a direct message.

Improving and Iterating

Order the problems you identify by frequency of them happening and their severity, and start fixing these issues. For the examples we outlined above, here are the correlated fixes:

  1. Two employees had a hard time entering the start date of their PTO in a way the bot can understand. Support multiple ways to enter a date. Use AI solutions if necessary.

  2. One employee started a PTO request in the channel rather than in a direct message. Create an error script that lets the user know that PTO requests only work in direct messages, and guide the users to start a direct message.

Once you’ve fixed these issues, run the test again with fresh test users, and enjoy the fact that you are making your bot better with each iteration.

This was an example of running a usability test to learn about and improve your bot before you launch it. There might be other types of tests you would like to run; ones that check user satisfaction or brand impact, for example. You might also use a different tool to prototype or create an alpha of your bot in order to run these tests. The most important thing is to iterate and learn.

Learning and iterating does not stop when you launch your bot in production—quite the opposite. Most bot builders I’ve spoken with have told me they have learned the most from looking at users using their bots in production, and collecting logs and analytics that they can use to constantly improve their bots. This will be the topic of our next chapter.

Continue reading How to design bot conversations.

Categories: Technology

120+ live online training courses opened for August and September

O'Reilly Radar - Wed, 2018/08/01 - 03:00

Get hands-on training in blockchain, machine learning, Python, Java, management, and many other topics.

Learn new topics and refine your skills with more than 120 live online training courses we opened up for August and September on our learning platform.

Space is limited and these courses often fill up.

Artificial intelligence and machine learning

Machine Learning in Python and Jupyter for Beginners, August 3

Artificial Intelligence: AI for Business, August 3

High Performance TensorFlow in Production: Hands on with GPUs and Kubernetes, August 14-15

Artificial Intelligence: Real-world Applications, August 30

Getting Started with Machine Learning, September 5

Artificial Intelligence: AI for Business, September 7

Managed Machine Learning Systems and Internet of Things, September 11-12

Deep Learning Fundamentals, September 12

Deploying Machine Learning Models to Production: A Toolkit for Real-World Success, September 13-14

Machine Learning in Python and Jupyter for Beginners, September 17

Artificial Intelligence: An Overview of AI and Machine Learning, September 21

Building a Deep Learning Model Using Tensorflow, September 26-27

Artificial Intelligence: Real-world Applications, September 27

Hands-on Machine Learning with Python: Classification and Regression, September 27

Machine Learning with Apache Spark 2.0x, September 28

Hands-on Machine Learning with Python: Clustering, Dimension Reduction, and Time Series Analysis, September 28


Understanding Hyperledger Fabric Blockchain, September 18-19

Blockchain Applications and Smart Contracts, September 20

Introducing Blockchain, September 28


Managing Your Manager, August 6

Introduction to Employee Performance Management, August 15

How to Give Great Presentations, August 21

Introduction to Time Management Skills, September 4

Understanding Business Strategy, September 7

Introduction to Lean, September 10

Introduction to Critical Thinking, September 10

Emotional Intelligence for Managers, September 12

Advanced Agile: Scaling in the Enterprise, September 12

Introduction to Customer Experience, September 18

Managing your Manager, September 21

Agile for Everybody, September 24

Your First 30 days as a Manager, September 24

Getting Started with Product Management, September 25

Leading Nimble and Adaptive Teams, September 28

Data science and data tools

Advanced SQL Series: Proximal and Linear Interpolations, August 6

Kafka Fundamentals, August 6-7

SQL Fundamentals for Data, August 7-8

Building Distributed Pipelines for Data Science Using Kafka, Spark, Cassandra, August 7-9

Digging Deeper with PostgreSQL: Automation with Scripts and Functions, and Postgres Inheritance, August 9

Digging Deeper with PostgreSQL: Organizing Processes, Data, and Evaluating Performance, August 10

Reactive Python for Data Science, August 13

Mastering Data Science at Enterprise Scale, August 22-23

Advanced SQL for Data Analysis (with Python, R, and Java), August 30

Python Data Handling: A Deeper Dive, September 4

Fundamental PostgreSQL, September 5-6

Analyzing and Visualizing Data with Microsoft Power BI, September 6

Digging Deeper with PostgreSQL: Organizing Processes, Data, and Evaluating Performance, September 10

Kafka Fundamentals, September 10-11

Apache Hadoop, Spark, and Big Data Foundations, September 11

Advanced SQL for Data Analysis (with Python, R, and Java), September 12

Getting Started with Pandas, September 13

Building Intelligent Bots in Python, September 13

Mastering Pandas, September 14

Real-Time Data Foundations: Kafka, September 18

Real-Time Data Foundations: Spark, September 19

Sentiment Analysis for Chatbots in Python, September 19

Modern Real-Time Streaming Architectures, September 19

Digging Deeper with PostgreSQL: Automation with Scripts and Functions, and Postgres Inheritance, September 19

SQL Fundamentals for Data, September 19-20

Hands-On Introduction to Apache Hadoop and Spark Programming, September 19-20

Real-Time Data Foundations: Flink, September 25

Real-Time Data Foundations: Time Series Architectures, September 26


Design Thinking: Practice and Measurement Essentials, September 5

Design Fundamentals for Non-Designers, September 14


Unit Testing with the Spock framework, August 1

Python Advanced: Generators, Coroutines, and Async/Await, August 1

Git Fundamentals, August 1-2

Functional Programming in Java, August 9-10

Getting Started with Spring and Spring Boot, August 13-14

Reactive Spring and Spring Boot, August 20

Unit Testing with the Spock Framework, August 23

Next-Generation Java Testing with JUnit 5, August 27

Java Testing with Mockito and the Hamcrest Matchers, August 30

Reactive Python for Data Science: Production-Ready, Scalable Code for Real-Time Data, September 4

Getting Started with Python's Pytest, September 4

Players Making Decisions: Decision-Making Frameworks for Game Designers, September 4

Programming with Java 8 Lambdas and Streams, September 4

Linux Performance Optimization, September 10

Consumer Driven Contracts: A Hands-On Guide to Spring Cloud Contract, September 11

JavaScript The Hard Parts: Closures, September 13

Getting Started with Python 3, September 13-14

Gradle: The Basics and Beyond, September 13-14

Building Chatbots with AWS, September 17

Getting Started with Java: From Core Concepts to Real Code in 4 Hours, September 17

Fundamentals of Virtual Reality Technology and User Experience, September 17

An Introduction to Go for Systems Programmers and Web Developers, September 17-18

Linux Troubleshooting, September 18

Python Programming Fundamentals, September 26


Threat Hunting in Practice, August 28

Introduction to Digital Forensics and Incident Response (DFIR), September 11

CompTIA Security+ SY0-501 Certification Practice Questions and Exam Strategies, September 12

Introduction to Ethical Hacking and Penetration Testing, September 17-18

CISSP Crash Course, September 17-18

Cybersecurity Offensive and Defensive Techniques in 3 Hours, September 19

Cyber Security Defense, September 28

Software architecture

From Monolith to Microservices, August 14-15

Implementing Evolutionary Architectures, September 3-4

AWS Certified Solutions Architect Associate Crash Course, September 4-5

From Developer to Software Architect, September 5-6

Google Cloud Platform Professional Cloud Architect Certification Crash Course, September 13-14

Shaping and Communicating Architectural Decisions, September 17

Systems engineering and operations

Creating serverless APIs with AWS Lambda and API Gateway, August 16

Applied Introduction to Amazon Web Services (AWS): Configure a Simple Web Application with Amazon VPC, EC2, and RDS, August 17

Amazon Web Services (AWS) Technical Essentials, August 27

Jenkins 2: Beyond the Basics, August 30

AWS Monitoring Strategies, August 31

Red Hat Certified System Administrator (RHCSA) Crash Course, September 4-7

9 Steps to Awesome with Kubernetes, September 5

Docker: Up and Running, September 5-6

AWS Certified SysOps Administrator (Associate) Crash Course, September 6-7

Google Cloud Certified Associate Cloud Engineer Crash Course, September 6-7

Getting Started with DevOps in 90 Minutes, September 10

Kubernetes in 3 Hours, September 11

Introduction to Google Cloud Platform, September 11-12

Building a Deployment Pipeline with Jenkins 2, September 12-13

Amazon Web Services (AWS) Technical Essentials, September 24

Getting Started with Continuous Integration (CI), September 24

Introduction to Kubernetes, September 24-25

Building and Managing Kubernetes Applications, September 28

Web programming

Beginning Frontend Development with React, August 2-3

Rethinking REST: A Hands-on Guide to GraphQL and Queryable APIs, September 4

Better Angular Applications with Observables: QuickStart, September 7

Component Driven Architecture in Angular, September 10

Angular Testing Quickstart, September 18

Beginning ASP.NET, September 20-21

Continue reading 120+ live online training courses opened for August and September.

Categories: Technology

The future of data storage lies in the foundation of biology

O'Reilly Radar - Wed, 2018/08/01 - 03:00

DNA holds the key to storing vast amounts of digital data

When we think of DNA, we inevitably think of genes and the biological instructions that our bodies follow to produce—well—us. DNA is a language composed of As, Ts, Gs, and Cs whose pattern dictates the characteristics that each and every one of us uniquely possess. But the interesting thing about DNA is that its data storage capability isn’t necessarily limited to biological uses.

In fact, what do deoxyribonucleic acid and magnetic tape have in common? They can store digital data in a similar way, albeit with different materials and therefore different outcomes.

Magnetic Tape as a Means of Storing Digital Data

Magnetic tape was first introduced in 1952 with IBM’s 762 tape. Since then, the size of magnetic tapes has drastically been reduced while still increasing storage capacities significantly. The 2012 Linear Tape-Open (LTO) 6, for example, took data storage up to 2.5 TB. Compare this with the original IBM 762 tape, which stored only 2 million digits and was 1,200 feet in length!

Figure 1-1. A timeline of the release and improvement of data-storing magnetic tape (Source: Iron Mountain)

The process of recording information on magnetic tape utilizes the binary system of 0s and 1s, which correspond to two magnetic polarities of the tape. A current-carrying conductor, known as the write head, can detect the magnetic quality of the tape and alter it according to the needed pattern. In contrast, the read head can produce current from the magnetic pattern, recording the binary code and thereby decoding the stored data.

Recording data on magnetic tape continues to be a viable option but two major issues exist with this form of storage:

  1. The world’s data is doubling every two years, making it difficult for magnetic tape production to keep up with such a large quantity of information

  2. According to studies, current magnetic tape has an attrition rate of 22% and only lasts 5 to 10 years before they must be replaced to preserve the stored data

The Difference in DNA Data Storage

There are currently 50 trillion trillion trillion base pairs of DNA on planet Earth; that’s 50 billion tons of DNA in total and good evidence as to the amount of information stored in this biological technology, considering that it comprises every living being.

DNA is a very dense form of storage. One gram of the substance could potentially hold a billion gigabytes of information, in a form factor that will well outlast any current data storage material. With evidence from the known fossil record, DNA provides a promising unit to hold information for hundreds of thousands of years, given the appropriate preservation conditions. For example, genetic material found in fossils of plants and insects recovered from Greenland ice cores dates to 800,000 years ago. The absence of oxygen and water mean that DNA can be better preserved and persist from ancient times to the contemporary world.

Natural methods of preservation imply positive results for digital storage in genetic material, but scientific advances in the glass encasing of DNA provide even greater hope for long-term use. The double-stranded integrity of DNA is preserved after being encapsulated by sol-gel derived silica, according to a paper by Kapusuz and Durucan. With novel methods to protect the biological matter, there is every reason to consider it as an alternative digital storage material.

However, the general approach of encoding information in DNA brings up a relevant issue: what is the cost of this type of data storage and how does it compare to the costs of current storage methods? According to synthetic DNA manufacturer, Twist Bioscience, the cost of storing information in DNA is equivalent to the costs of the reagents needed to add additional bases to DNA molecules. Twist’s estimation of these costs is “less than a fraction of a penny per gigabyte to encode and recover stored information” (DNA-Based Digital Storage). Assuming the synthetic DNA would possess a longer shelf-life, the cost of this data storage method is cheaper than the alternative magnetic tape option.

Of course, the cost of sequencing or “reading” the DNA must also be factored in. The human genome project from 1990–2003 required roughly $3 billion to sequence the entire human genome, about 4 GB of information. In 2014, the same process could be done for about $1,000 due to Illumina’s whole-genome sequencing efforts. That represents a huge cost reduction, which provides hope for greater affordability in the future, despite the fact that storing 1 TB of information on a disc drive currently costs $100.

Figure 1-2. A simplification of the process of long-term DNA data storage (Source: Twist Bioscience)

The process of storing information in DNA is straightforward in that the binary code traditionally used in information technology is converted to the equivalent adenine, thymine, guanine, and cytosine nucleotides that make up genetic material. This conversion is acquired through algorithms from telecommunications and other information technologies, according to Twist Bioscience. The company provides the following example of a conversion between binary code and nucleotide bases: “00 represents A, 10 represents C, 01 represents G, and 11 represents T.” Increasing location choices from two (0 or 1) to four (A, T, G, or C) is an indication of an increase in efficiency from Twist’s perspective.

But what about errors in media transmissions? Algorithms known as error correction codes have been developed to prevent error transmissions in the transition between base two and base four codes, that is between binary code and DNA bases. The error correction codes are the primary reason for the short length of approximately 100 to 200 bytes of the then synthesized DNA strands. These chunks of DNA must be ordered correctly with the help of “biological barcodes” that are created at the ends of the synthesized segments (Twist Bioscience).

To determine the success of the encoding process, these strands of DNA can undergo reverse processes, which return the genetic material to binary code that can be read as the original digital information. This involves isolating the DNA and amplifying it through a common laboratory technique, the polymerase chain reaction (PCR). The DNA can then be sequenced in short chunks and once the base pattern is established, it can be reconverted to binary code and pieced together to obtain the initial digital information. Because a portion of the DNA is taken for sequencing, each strand must be duplicated through PCR to ensure that the original data is still available, even after it is decoded.

The time is takes to decode information in DNA is another measure of its feasibility for use as a storage medium. In 2014, the human genome could be sequenced at a rate of approximately 1 GB per hour. At the same time, Illumina could sequence 18,000 human genomes per year, translating to about 72 TB of information (to put 72 TB into perspective, that’s about 24 disc drives, each roughly the size of a book, with 3TB of storage per drive).

With the advent of new technology, again by Illumina, 4 GB of DNA sequencing can now be accomplished in one hour!

Current Advances in DNA Data Storage

DNA storage is not a new concept. In 2012, scientist George Church and colleagues at Harvard University encoded Church’s book Regenesis: How Synthetic Biology Will Reinvent Nature and Ourselves.

Since then, other advancements have been made in technology which have established genetic material as an even more relevant material for digital information storage. In a recent partnership between Twist Bioscience, Microsoft, and the University of Washington, performances of two musical pieces at the Montreux Jazz Festival have been successfully stored in DNA using the process described above.

DNA storage is meant to condense and replace magnetic tape storage, miniaturizing everything in the process and therefore saving space for the large quantities of data we produce. Microsoft is thought to have the goal of including DNA in the cloud as well as having genetic material in data centers for operational storage.

Besides the storage longevity and compactness of DNA, there remains an important point about the medium. Deoxyribonucleic acid is a substance from nature itself. Technology is headed in the direction of biomimicry and therefore what storage element would be more fitting to house the vast quantities of digital information we create than the storage capacity that supports virtually all life on Earth?


Continue reading The future of data storage lies in the foundation of biology .

Categories: Technology

Four short links: 1 August 2018

O'Reilly Radar - Wed, 2018/08/01 - 01:00

Data Science Ethics, Bandit Algorithms, Formal Methods, and FAST Goals

  1. Data's Day of Reckoning (Loukides, Mason, Patil) -- Data science, machine learning, artificial intelligence, and related technologies are now facing a day of reckoning. It is time for us to take responsibility for our creations. What does it mean to take responsibility for building, maintaining, and managing data, technologies, and services?
  2. Bandit Algorithms (Tor Lattimore) -- A practitioner seeking to apply a bandit algorithm must understand which assumptions in the theory are important and how to modify the algorithm when the assumptions change. We hope this book can provide that understanding. Bandit algorithms make decisions with partial information, taking into account the cost of getting more information.
  3. Augmenting Agile with Formal Methods -- The difference between writing TLA+ and just writing unit tests isn’t half an hour versus sixteen hours, it’s half an hour versus “Two weeks to realize there’s a bug, a week to find the bug, three days to understand the bug, sixteen hours to write the test, twenty minutes to run the test, and you don’t know if your fix really works.”
  4. FAST Goals Beat SMART Goals (MIT Sloan Review) -- FAST = Frequently-discussed, Ambitious, Specific, and Transparent. (via Helen Bevan)

Continue reading Four short links: 1 August 2018.

Categories: Technology

Data's day of reckoning

O'Reilly Radar - Tue, 2018/07/31 - 04:00

We can build a future we want to live in, or we can build a nightmare. The choice is up to us.

Our lives are bathed in data: from recommendations about whom to “follow” or “friend” to data-driven autonomous vehicles. But in the past few years, it has become clear that the products and technologies we have created have been weaponized and used against us. Although we’ve benefited from the use of data in countless ways, it has also created a tension between individual privacy, public good, and corporate profits. Cathy O’Neil’s Weapons of Math Destruction and Virginia Eubanks’ Automating Inequality document the many ways that data has been used to harm the broader population.

Data science, machine learning, artificial intelligence, and related technologies are now facing a day of reckoning. It is time for us to take responsibility for our creations. What does it mean to take responsibility for building, maintaining, and managing data, technologies, and services? Responsibility is inevitably tangled with the complex incentives that surround the creation of any product. These incentives have been front and center in the conversations around the roles that social networks have played in the 2016 U.S. elections, recruitment of terrorists, and online harassment. It has become very clear that the incentives of the organizations that build and own data products haven’t aligned with the good of the people using those products.

These issues aren’t new to the consumer internet. Other fields have had their days of reckoning. In medicine, widely publicized abuses include the Tuskegee syphilis experiment, the case of Henrietta Lacks (whose cells were used for cancer research without her permission and without compensation), and human experiments performed during World War II by Nazis. The physics community had to grapple with the implications of the atomic bomb. Chemists and biologists have had to address the use of their research for chemical and biological weapons. Other engineering disciplines have realized that shoddy work has an impact on people’s lives; it’s hard to ignore bridge collapses. As a result, professional societies were formed to maintain and enforce codes of conduct; government regulatory processes have established standards and penalties for work that is detrimental to society.

Ethics and security training

In many fields, ethics is an essential part of professional education. This isn’t true in computer science, data science, artificial intelligence, or any related field. While courses on ethics exist at many schools, the ideas taught in ethics classes often aren’t connected to existing projects or course work. Students may study ethical principles, but they don’t learn how to implement those principles in their projects. As a result, they are ill-prepared for the challenges of the real world. They’re not trained to think about ethical issues and how they affect design choices. They don’t know how to have discussions about projects or technologies that may cause real-world harm.

Software security and ethics frequently go hand in hand, and our current practices for teaching security provide an example of what not to do. Security is usually taught as an elective, isolated from other classes about software development. For example, a class on databases may never discuss SQL injection attacks. SQL injection would be addressed in classes on security, but not in the required database course. When a student submits a project in a database course, its vulnerability to hostile attack doesn’t affect the grade; an automated grading system won’t even test it for vulnerabilities. Furthermore, a database course might not discuss architectural decisions that limit damage if an attacker gains access—for example, storing data elements such as names and social security numbers in different databases.

Teaching security in an elective is better than not teaching it at all; but the the best way to produce programmers who really understand security is to incorporate it into assignments and grading within the core curriculum, in addition to teaching it in electives. The core curriculum ensures that everyone can recognize and deal with basic security problems; the electives can go into greater depth, and tie together issues from different disciplines ranging from physical security to cryptography. Security as an afterthought doesn’t work in product development; why do we expect it to work in education? There is no industry in which security lapses haven’t led to stolen data, affecting millions of individuals. Poor security practices have led to serious vulnerabilities in many consumer devices, from smart locks to smart light bulbs.

Ethics faces the same problem. Data ethics is taught at many colleges and universities, but it’s isolated from the rest of the curriculum. Courses in ethics help students think seriously about issues, but can’t address questions like getting informed consent in the context of a real-world application. The White House report "Preparing for the Future of Artificial Intelligence" highlights the need for training in both ethics and security:

Ethical training for AI practitioners and students is a necessary part of the solution. Ideally, every student learning AI, computer science, or data science would be exposed to curriculum and discussion on related ethics and security topics. However, ethics alone is not sufficient. Ethics can help practitioners understand their responsibilities to all stakeholders, but ethical training should be augmented with technical tools and methods for putting good intentions into practice by doing the technical work needed to prevent unacceptable outcomes.

Ethics and security must be at the heart of the curriculum, not only as electives, or even isolated requirements. They must be integrated into every course at colleges, universities, online courses, and programming boot camps. They can’t remain abstract, but need to be coupled with “technical tools and methods for putting good intentions into practice.” And training can’t stop upon graduation. Employers need to host regular forums and offer refresher courses to keep people up-to-date on the latest challenges and perspectives.

Developing guiding principles

The problem with ethical principles is that it’s easy to forget about them when you’re rushing: when you’re trying to get a project finished on a tight, perhaps unrealistic, schedule. When the clock is ticking away toward a deadline, it’s all too easy to forget everything you learned in class—even if that class connected ethics with
solutions to real-world problems.

Checklists are a proven way to solve this problem. A checklist, as described by Atul Gawande in The Checklist Manifesto, becomes part of the ritual. It’s a short set of questions that you ask at the start of the project, and at every stage as you move toward release. You don’t go to the next stage until you’ve answered all the questions affirmatively. Checklists have been shown to reduce mistakes in surgery; they’re used very heavily by airline pilots, especially in emergencies; and they can help data professionals to not forget ethical issues, even when they are under pressure to deliver.

Here’s a checklist we’ve developed for developers working on data-driven applications:

❏ Have we listed how this technology can be attacked or abused?

❏ Have we tested our training data to ensure it is fair and representative?

❏ Have we studied and understood possible sources of bias in our data?

❏ Does our team reflect diversity of opinions, backgrounds, and kinds of thought?

❏ What kind of user consent do we need to collect and use the data?

❏ Do we have a mechanism for gathering consent from users?

❏ Have we explained clearly what users are consenting to?

❏ Do we have a mechanism for redress if people are harmed by the results?

❏ Can we shut down this software in production if it is behaving badly?

❏ Have we tested for fairness with respect to different user groups?

❏ Have we tested for disparate error rates among different user groups?

❏ Do we test and monitor for model drift to ensure our software remains fair over time?

❏ Do we have a plan to protect and secure user data?

Feel free to modify this checklist to fit your situation and use it in your projects.

The Fairness, Accountability, and Transparency in Machine Learning group (FAT/ML) advocates a similar approach. Their Principles for Accountable Algorithms and a Social Impact Statement for Algorithms suggests assessing the social impact statement of a project at least three times during its life: during design, pre-launch, and post-launch. Working through a social impact statement requires developers to think about the ethical consequences of their projects and address any problems that turn up. In a similar vein, the Community Principles on Ethical Data Practices, which arose out of the Data For Good Exchange (D4GX), provides a set of values and principles that have been gathered through community discussion. They’re a great start for any group that wants to create its own checklist. And Cathy O’Neil has proposed auditing machine learning algorithms for fairness.

Building ethics into a data-driven culture

Individual responsibility isn’t sufficient. Ethics needs to be part of an organization’s culture. We’ve seen many organizations recognize the value of developing a data-driven culture; we need to ensure ethics and security become part of that culture, too.

Security is gradually becoming a part of corporate culture: the professional, financial, legal, and reputational consequences of being a victim are too large to ignore. Organizations are experimenting with bug-bounty programs, sharing threats with each other, and collaborating with government agencies. Security teams are no longer simply corporate naysayers; they’re charged with preventing serious damage to an organization’s reputation and to finances.

Integrating ethics into corporate culture has been more challenging. A single team member may object to an approach, but it’s easy for an individual to be overruled, and if there’s no support for ethical thinking within the organization, that’s likely to be where it ends. Ethical thinking is important with or without corporate support, but it’s more likely to make a difference when ethical action is a corporate value. Here are some ideas for building ethics into culture:

  • An individual needs to be empowered to stop the process before damage is done. Toyota and W. Edwards Deming pioneered the use of the andon cord to improve quality and efficiency. Anyone who saw a problem could pull the cord, which would halt the production line. Senior managers as well as production line operators would then discuss the issue, make improvements, and restart the process.

    Any member of a data team should be able to pull a virtual “andon cord,” stopping production, whenever they see an issue. The product or feature stays offline until the team has a resolution. This way, an iterative process can be developed that avoids glossing over issues.
  • Anyone should be able to escalate issues for remediation without fear of retaliation. There needs to be an escalation process for team members who don’t feel their voice has been heard. The U.S. Department of State has a dissent channel where any diplomat can make sure the Secretary of State hears their concerns. In health care, a path to escalate legal and ethical issues is required by law. For health care plans in the U.S., there is a compliance officer who reports directly to the board of directors.

    Data-driven organizations need a similar model that allows people to escalate issues without the fear of reprisal. An escalation process could be implemented in several forms. For example, companies could work with an organization such as the Electronic Frontier Foundation (EFF) to develop a program that accepts and investigates whistleblower reports. The problem would be kept from public scrutiny unless specific criteria are violated. A similar approach could be implemented under an existing or new agency (e.g., a Consumer Data Protection Agency).
  • An ethical challenge should be part of the hiring process. When hiring, companies frequently assess whether a candidate will be a "cultural fit." Interviewers ask questions that help them understand whether a candidate will work well with other team members. However, interviewers rarely ask questions about the candidate’s ethical values.

    Rather than asking a question with a right/wrong answer, we’ve found that it’s best to pose a problem that lets us see how the candidate thinks about ethical and security choices. Here’s a question we have used:

    Assume we have a large set of demographic data. We’re trying to evaluate individuals and we’re not supposed to use race as an input. However, you discover a proxy for race with the other variables. What would you do?

    This kind of question can start a dialogue about how to use the proxy variable. What effects does it have on people using the product? Are we making recommendations, or deciding whether to provide services? Are we implementing a legal requirement, or providing guidance about compliance? Discussing the question and possible answers will reveal the candidate’s values.
  • Product reviews must ask questions about the product’s impact. Environmental impact statements predict the impact of construction projects on the public. We’ve already mentioned FAT/ML’s proposed Social Impact Statements as an example of what might be done for data. In the social sciences and the biomedical industry, Institutional Review Boards (IRBs) assess the possible consequences of experiments before they’re performed.

    While both environmental impact statements and IRBs present problems for data products, data teams need to evaluate the impact of choices they make. Teams need to think about the consequences of their actions before releasing products. We believe that using a checklist is the best approach for ensuring good outcomes.
  • Teams must reflect diversity of thought, experiences, race, and background. All too often, we hear about products that are culturally insensitive or overtly racist. One notorious example is an automated passport control system that doesn’t let an individual proceed until a good digital image is captured. People of Asian ancestry reported that the system kept asking them to open their eyes, even though their eyes were open. Many cringe-worthy examples are well documented; they can often be traced to a lack of data or a lack of insight into the diversity of the population that will be impacted.

    While there’s no general solution to these problems of cultural sensitivity, diversity and inclusion are a tremendous help. Team members should be from the populations that will be impacted. They’ll see issues well before anyone else. External peer reviews can help to reveal ethical issues that your team can’t see. When you’re deeply involved with a project, it can be hard to recognize problems that are obvious to outsiders.
  • Corporations must make their own principles clear. Google’s “Don’t be evil” has always been a cute, but vague, maxim. Their recent statement, Artificial Intelligence at Google: Our Principles, is much more specific. In a similar vein, the face recognition startup Kairos has said that they won’t do business with law enforcement companies. Kairos’ CEO writes that “the use of commercial face recognition in law enforcement or government surveillance of any kind is wrong.”

    However, it’s important to realize that advocating for corporate ethical principles has consequences. Significant internal protest, and the resignation of several developers in protest over Google’s defense contracts, were needed to get their AI principles in place. Kairos is probably leaving a lot of money on the table. It’s also important to realize that organizations frequently point to their ethical principles to divert attention from unethical projects.

Over the past few years, we’ve heard a lot about software startups that begin with a “minimal viable product,” and adhere to Facebook’s slogan, “move fast and break things.” Is that incompatible with the approach we’ve just described? This is a false choice. Going fast doesn’t mean breaking things. It is possible to build quickly and responsibly.

The lean/agile methodology used in many startups is a good way to expose ethical issues before they become problems. Developers start with a very simple product (“the simplest thing that could possibly work,” according to Ward Cunningham’s seminal phrase), demo it to users, get feedback, develop the next version, and repeat. The process continues for as many iterations as needed to get a product that’s satisfactory. If a diverse group of users tests the product, the product development loop is likely to flush out systematic problems with bias and cultural insensitivity. The key is testing the product on a truly diverse group of users, not just a group that mirrors the expected customer base or the developers’ backgrounds.


In some industries, ethical standards have been imposed by law and regulation. The Nuremberg Code was developed in response to Nazi atrocities. It focuses on individual consent to participation in an experiment. After the Tuskegee syphilis experiments became public knowledge, the code was put into law in the 1974 National Research Act and the 1975 Declaration of Helsinki. This push to codify ethical guidelines established the role of the institutional review board (IRB), and was adopted widely in the U.S via the Common Rule.

In other industries, other regulatory bodies enforce ethical standards These include the U.S. Federal Trade Commission (FTC), which oversees commerce; the Nuclear Regulatory Commission (NRC), which oversees nuclear power plants; the Federal Food and Drug Administration (FDA), which oversees the safety of pharmaceuticals; and, most recently, the Consumer Finance Protection Bureau (CFPB), which oversees bankers and lenders on behalf of consumers.

The European Union’s General Data Protection Regulation (GDPR) takes an aggressive approach to regulating data use and establishing a uniform data policy. In June 2018, California passed a digital privacy law similar to GDPR, despite the reservations of many online companies. One challenge of developing a policy framework is that the policy development process nearly always lags the pace of innovation, and isn’t agile enough to keep policy iterative. By the time a policy has been formulated and approved, it almost always lags behind technology; but it’s impossible for policy makers to iterate quickly enough to catch up with the newest technology. Another problem is that the committees that make policy often lack experts with the necessary technical background. That can be good; technologists are too easily influenced by “what technology wants.” But policies created by people who are technologically uninformed are frequently out of touch with reality: look at the debate over back doors to encryption protocols.

Some have argued that organizations using data should adopt the Institutional Review Board (IRB) model from the biomedical industry. Unfortunately, while there are many positive aspects of the IRB, this isn’t a viable approach. IRBs are complex, and they can’t be agile; it’s very difficult for IRBs to adapt to new ideas and technologies. It’s why the Obama Administration pushed for nearly eight years to update the Common Rule’s models for consent to be consistent with digital technologies and to enable data mining.

Building our future

For some time, we’ve been aware of the ethical problems that arise from the use and abuse of data. Public outcry over Facebook will die down eventually, but the problems won’t. We’re looking at a future in which most vehicles are autonomous; we will be talking to robots with voices and speech patterns that are indistinguishable from humans; and where devices are listening to all our conversations, ready to make helpful suggestions about everything from restaurants and recipes to medical procedures. The results could be wonderful—or they could be a nightmarish dystopia.

It’s data’s day of reckoning. The shape of the future will depend a lot on what we do in the next few years. We need to incorporate ethics into all aspects of technical education and corporate culture; we need to give people the freedom to stop production if necessary, and to escalate concerns if they’re not addressed; we need to incorporate diversity and ethics into hiring decisions; and we may need to consider regulation to protect the interests of individual users, and society as a whole.

Above all, talk about ethics! In "It’s time for data ethics conversations at the dinner table," Natalie Evans Harris and others write that “we need to be having difficult conversations about our individual and collective responsibility to handle data ethically.” This is the best single thing you can do to further data ethics: talk about it in meetings, at lunch, and even at dinner. Signing a data oath, or agreeing to a code of conduct, does little if you don’t live and breathe ethics. Once you are living and breathing it, you will start to think differently about the code you write, the models you build, and the applications you create. The only way to create an ethical culture is to live it. The change won’t take place magically, nor will it be easy—but it’s necessary.

We can build a future we want to live in, or we can build a nightmare. The choice is up to us.

Continue reading Data's day of reckoning.

Categories: Technology

Four short links: 31 July 2018

O'Reilly Radar - Tue, 2018/07/31 - 01:00

Quantum Circuits, Site Reliability, AI and Board Games, and Designing for Use

  1. Quirk -- a drag-and-drop quantum circuit simulator. (via Hacker News)
  2. Site Reliability Workbook -- how to implement SRE at your org, available online for free until August 23 or always from Amazon.
  3. Blood Bowl: The Next Board Game Challenge for AI -- At first sight, the game ought to be approachable by numerous game-playing algorithms. However, as all pieces on the board belonging to a player can be moved several times each turn, the turn-wise branching factor becomes overwhelming for traditional algorithms. Additionally, scoring points in the game is rare and difficult, which makes it hard to design heuristics for search algorithms or apply reinforcement learning.
  4. How We Redesigned Dropbox for Rapid Mobile Work -- You’re no longer designing to extend engagement or keep customers in the app. Rather, you’re helping people get in and out and done. Please, more design for this goal.

Continue reading Four short links: 31 July 2018.

Categories: Technology

How companies can get started with AI

O'Reilly Radar - Mon, 2018/07/30 - 04:00

The program for our Artificial Intelligence Conference in London is structured to help companies that are still very much in the early stages of AI adoption.

Judging by the list of countries putting out policy papers on AI and automation technologies, there is very strong interest in AI across the globe. In order to asses the current state of readiness across regions, we recently conducted a survey (full report forthcoming) of the state of adoption of machine learning tools and technologies (a lot of what is being currently described as AI is really ML). The survey yielded 11,400+ respondents, including 2,000 respondents from Europe:

As we assembled the program for our inaugural Artificial Intelligence Conference in London this October, we recognized that many companies and organizations around the world are still very much in the early stages of adoption. Anyone wanting to get started on AI technologies will have to wade through an array of methods and technologies, many of which are still very much on the leading edge. The good news is that some companies at the forefront are beginning to share best practices, tools, and lessons learned as they deploy AI technologies. In this short post, I’ll use key portions of our AI conference in London to describe how companies may want to get started in AI.

AI in the Enterprise: Best practices

Has machine learning impacted how companies organize and build teams? We found that as companies gain more experience with machine learning, they are more likely to start hiring and attracting specialists, including data scientists, research scientists, and machine learning engineers.

But there are many more decisions companies need to make on the path to embracing AI. Organizations must work through a series of assessments in order to successfully deploy AI and automation technologies:

  • What are the key technologies (hardware and software) and what are their limitations?
  • How much data does one need to use these technologies? Where can I get data to augment my existing data sets?
  • How does one organize and hire for AI?
  • What types of problems or tasks lend themselves to automation? What are some initial use cases one can consider?
  • How do you maintain momentum and build upon the lessons learned from previous projects?

We put together training programs, tutorials, and sessions designed to help attendees understand how to move forward with best practices, tools, and technologies used at many leading companies.

Early applications of AI technologies

As I noted earlier, much of the current excitement around AI pertains to recent progress in machine learning—specifically deep learning and reinforcement learning. Both of these class of methods impact existing products and data types (text, temporal, and structured data) and also enable companies to unlock new data sources (including audio, video, and images). Progress in automating enterprise workflows will depend on foundational technologies (hardware and software) and breakthroughs in areas like computer vision, natural language understanding and generation, and speech technologies. We are beginning to see many interesting enterprise applications of both deep learning and reinforcement learning, particularly in computer vision and text, but also in areas where many enterprises already had analytic solutions in place (recommenders and forecasting tools).

Implementing AI

AI applications rely on machine learning models, as well as hardware and software infrastructure for model training, deployment, and management. Machine learning itself requires robust end-to-end data pipelines spanning data ingestion, data preparation, and data management. Depending on the nature of the application, a knowledge base or graph, components for reasoning and planning, simulation platforms, and user interfaces might also come into play. For our upcoming AI conference in London, we assembled sessions on many of core components in an AI technology stack. We have content ranging from tutorials on how to use specific libraries and technologies to how to launch data markets and networks to sessions on best practices for building and architecting AI applications.

Case studies

Among the many the challenges faced by organizations is identifying good use cases for AI and automation technologies. One of our main goals for this conference series is to foster a community of professionals interested in building and using AI applications. To that end, we put together a series of sessions where companies describe how they put AI technologies to work within their organizations. We are also planning a series of attendee networking events at the conference. Here’s a sampling of sessions from a few domains:

Ethics, privacy, security

As I noted in a recent post, there is growing awareness among major stakeholders about the importance of data privacy, ethics, and security. In our recent ML adoption survey, we found that respondents are starting to engage and they are beginning to take into account factors such as bias, transparency, and privacy in their machine learning systems. Fairness and transparency, and privacy-preserving analytics have become areas of active academic research, but it’s important to emphasize that tools and best practices are just beginning to emerge. These concerns cut across geographic regions and industries, but with the launch of GDPR, Europe stands out for taking a leadership position in data privacy. AI policy papers from several European countries also emphasize the need for fairness and transparency in machine learning and AI. With all these considerations in mind, for our inaugural Artificial Intelligence Conference in London we have assembled an outstanding series of presentations on the practical aspects of incorporating ethics, privacy, and security into AI systems.

Continue reading How companies can get started with AI.

Categories: Technology


Subscribe to LuftHans aggregator - Technology