You are here

Feed aggregator

Four short links: 24 October 2018

O'Reilly Radar - Wed, 2018/10/24 - 04:00

Good Modeling, Real-Time Command Line, Data Structure Synthesis, and Free the Law

  1. About Our Model -- high school stats class makes a model for election prediction and lists all their assumptions. A good role model for the rest of us.
  2. Ultimate Plumber -- a tool for writing Linux pipes in a terminal-based UI interactively, with instant live preview of command results. Real-time feedback changes the command line.
  3. Generalized Data Structure Synthesis -- This paper shows how to synthesize data structures that track subsets and aggregations of multiple related collections. Our technique decomposes the synthesis task into alternating steps of query synthesis and incrementalization. [...] As an added benefit of this approach over previous work, the synthesized data structure is optimized for not only the queries in the specification but also the required update operations. I love tools that automate or expedite programming.
  4. Appeals Court to Georgia: Code Can't Be Copyrighted (EFF) -- On Friday, the U.S. Court of Appeals for the 11th Circuit handed down a powerful opinion that struck down the state of Georgia’s attempt to use copyright to suppress publication of its own laws. The ruling, which gives Georgians the right to read and publish the Official Code of Georgia Annotated, or OCGA, may also improve public access to legislative documents in other states. Georgia claimed copyright on their own laws and charged citizens to read them, bringing suit against Public Resource for publishing those laws for free.

Continue reading Four short links: 24 October 2018.

Categories: Technology

Four short links: 23 October 2018

O'Reilly Radar - Tue, 2018/10/23 - 04:10

Forecasting, Cars and Privacy, Quantum Communications, and Positive Communications

  1. Forecasting at Uber: An Introduction -- Actually, classical and ML methods are not that different from each other, but distinguished by whether the models are more simple and interpretable or more complex and flexible. In practice, classical statistical algorithms tend to be much quicker and easier-to-use. An important message that isn't getting as much airplay as the sales pitches: deep learning is unfairly good on some problems, but not all.
  2. The Next Data Minefield Is Your Car -- GM captured minuted details such as station selection, volume level, and ZIP codes of vehicle owners, and then used the car’s built-in Wi-Fi signal to upload the data to its servers. The goal was to determine the relationship between what drivers listen to and what they buy and then turn around and sell the data to advertisers and radio operators. And it got really specific: GM tracked a driver listening to country music who stopped at a Tim Horton’s restaurant. (No data on that donut order, though.)
  3. Inside Europe's Quest for an Unhackable Quantum Internet (MIT TR) -- China has also built a land-based QKD communications network from Beijing to Shanghai that banks and other companies are using to transmit sensitive commercial data. China's approach requires trusted quantum-classical-quantum repeaters every 10km, whereas the Dutch university at the center of this article is looking to use quantum teleportation. Interesting to see the Dutch are connecting universities, as ARPA did at the birth of the internet.
  4. GNU Kind Communications Guidelines -- astonishingly useful set of specific and positive recommendations. No mention of consequences of violation.

Continue reading Four short links: 23 October 2018.

Categories: Technology

How architecture evolves into strategy

O'Reilly Radar - Tue, 2018/10/23 - 04:00

A look at the roles of architect and strategist, and how they help develop successful technology strategies for business.

There are two jobs in the world that people want to do the most while knowing the least about: architect and strategist.

I should start by saying this section does not offer a treatise on how to do architecture. I'm offering an overview of my perspective on the field, which I hope is a unique and interesting take on it, in order to provide context for the work at hand: devising a winning technology strategy for your business.

Technology systems are difficult to wrangle. Our systems grow in accidental complexity and complication over time. Sometimes we can succumb to thinking other people really hold the cards, that they have the puppet strings we don't.

This is exacerbated by the fact that our field is young and growing and changing, and we're still finding the roles we need to have to be successful. To do so, we borrow metaphors from roles in other industries. The term "data scientist" was first used in the late 1990s. In 2008 or so, when data scientist emerged as a job title, it was widely ridiculed as a nonjob: the thought that people who just worked with data could be scientists, or employ the rigors of their time-honored methods, was literally laughable in many circles. By 2012, Harvard Business Review published an article by Thomas Davenport and D.J. Patil called "Data Scientist: The Sexiest Job of the 21st Century." Today, it's one of the most desired jobs, with pundits declaiming the terrifying state that we do not have nearly enough of them to tackle our most central technology problems.

Likewise, the term "architect" didn't enter popular usage to describe a role in the software field until the late 1990s. It, too, was ridiculed as an overblown, fancy-pants misappropriation from a "real" field. Part of the vulnerability here is that it hasn't always been clear what the architect's deliverables are. We often say "blueprints," but that's another metaphor borrowed from the original field, and of course we don't make actual blueprints.

With such origins, and with the subsequent division of the architect role into enterprise architect, solution architect, data architect, and so forth, the lines have blurred further. The result is that decades later, the practice and the art of the architect in technology varies dramatically not only from one company to the next but also from one department and one practitioner to the next.

So, we will define the role of the architect in order to proceed from common ground. This is my tailored view of it; others will have different definitions. Before we do that, though, let's cover some historical context that informs how we think of the role.

Vitruvius and the principles of architecture

Architecture begins when someone has a nontrivial problem to be solved. The product management team states what must be done to solve the problem, and the architect describes how to realize that vision in a system.

The first architect of record is a fellow named Vitruvius, who worked as a civil engineer in Rome in the first century B.C. While you may not know his name, during the Renaissance, Leonardo da Vinci popularized the "Vitruvian Man" with perfect proportions based on Vitruvius's ideas. Everyone who goes to architecture school learns his work.

Vitruvius is the author of de Architectura, known today as Ten Books on Architecture. It's a delightful, engaging read and had a strong influence on Renaissance artists such as Michelangelo and da Vinci. In it, Vitruvius expands on the three requirements any architecture must demonstrate:

It must be solid, firm.
It must be useful, have utility.
It must be beautiful, like Venus, inspiring love. This is sometimes translated as "delightful."

It's a given that we must design a system, including a local software architecture, that actually runs, that is "solid." It may need to run for many years, even decades, and be maintainable to adapt to changes over that time. Solid doesn't mean inflexible. Skyscrapers are built on purpose to sway slightly with the wind, specifically to be more durable. The Sears Tower in Chicago regularly sways between six inches and a foot; taller buildings in America sway as much as four to five feet. Your architectures, and your strategies, must be similarly flexible in order to endure. We'll look at this later when we discuss how to support evolutionary architectures through our strategies.

It must also be fit to purpose, which means understanding deeply what the real purpose of the system is and how to manage user expectations. This is supported in real terms through standards and consistent application of conventions, both in the information architecture (i.e., the user experience and design) and within the software construction itself.

Beauty, for Vitruvius, isn't really in the eye of the beholder. It is about harmony of proportion. One suggestion we can deduce from this for our current purposes is that we must rightsize our architecture and strategy work for the task at hand.

Vitruvius states—without irony—that an architect must concern himself with and become educated in several diverse fields of study, such that they find their way into the work. He outlines them in Chapter 1 of de Architectura:

  • Skill in manual labor as well as in theory

  • Proclivity and desire for continuous learning

  • A dexterity with tools

  • An understanding of optics—how the light gets in

  • History, such that you can emphasize and not misinterpret signs of cultural significance

  • A strong understanding of philosophy, in order to practice abstract thinking as well as honesty and courtesy

  • Physics, to help make things sturdy

  • Art, music, theater, drawing, painting, and poetry, to help make things beautiful and well suited to their human purposes

  • Math

  • Medicine

  • Astronomy

  • Politics

He concludes that absent a degree of education and even lay practice in any one of these areas, one cannot refer to oneself as an architect. These are excellent guides for us in technology today. For those of us concerned with the business of making software and setting the direction for other technologists, to hold ourselves to account in these ways would serve us very well.

In a recent conversation I had with Ben Pring, philosopher, noted futurist, and director of The Future of Work Center at Cognizant, he underscored the importance of beauty in software, pointing out that historically our most culturally significant buildings have been not merely adorned but specifically built with beauty in mind as a central, driving narrative. I conclude from this that such foregrounding reinforces in the popular imagination the power of the institutions that build them. I base this conclusion on the preface in the Ten Books, in which Vitruvius writes openly and directly to Emperor Caesar, stating:

But when I saw that you were giving your attention not only to the welfare of society in general and to the establishment of public order, but also to the providing of public buildings intended for utilitarian purposes, so that not only should the State have been enriched with provinces by your means, but that the greatness of its power might likewise be attended with distinguished authority in its public buildings, I thought that I ought to take the first opportunity to lay before you my writings on this theme. (emphasis mine)

Realizing these broad dicta into an architecture means, I think, finding the concentrations of power, and determining how to best support and ultimately inspire the human factor in the forms we create. I hope once you're done with this book, you'll have some ideas for how to enable and reveal the three facets of firmitas, utilitas, and venustas in your own work.

Three concerns of the architect

Whereas developers are typically focused on delivering working code for a user story within the next two weeks for one system within their one team, architects are concerned with how technology can fulfill business goals given a long-term outlook across a variety of interrelated systems across many teams. It's analogous to a project view versus a portfolio view. They should have their visors raised much higher. The architect is hopefully not concerned with low-level details of the code itself inside one system but is more focused on where data center boundaries are crossed, where system component boundaries are crossed.

Here's my definition of an architect's work: it comprises the set of strategic and technical models that create a context for position (capabilities), velocity (directedness, ability to adjust), and potential (relations) to harmonize strategic business and technology goals. Notice that in this definition, the role of the architect and technology strategist is not to merely serve the business but to play together. I have been in shops where technology was squarely second fiddle, a subservient order-taking organization to support what was deemed the real business. That's no fun for creative people who have something to contribute. But more importantly, I submit that businesses, now more than ever, cannot sustain such a division and to create greater competitive advantage must work toward integration with co-leadership.

Over my 20 years in this field, I've come to conclude that there are three primary concerns of the architect:

  • Contain entropy.

  • Specify the nonfunctional requirements.

  • Determine trade-offs.

There are many different roles that architects legitimately play in organizations. But the primary struggle I have seen comes when they are not focused on a deliverable, on what could be conceived as a "blueprint." Without that focus, they tend to weigh in at project meetings or make declarations informally that can't be remembered or followed. To stay pertinent to the project, and to help guide it in a way that others may not have the purview to do, drawing a line at these boundaries seems to work out pretty well. The definition remains, of course, rather open to interpretation, in grudging deference to the machinations of the real world.

Let's unpack each of those responsibilities.

Contain entropy

This viewpoint on the architect's work I learned in a fun conversation over dinner in New York with the very smart and funny Cameron Purdy, the founder of Coherence, who at the time ran Java at Oracle. "Entropy" refers to the second law of thermodynamics, which roughly states that systems over time will degrade into an increasingly chaotic state, such that the amount of energy in the system available for work is diminished.

The architect defines standards, conventions, and tool sets for teams to use. These are common practices and are generally idiosyncratic to any given organization. As application or solution architects, they help within a system, within an ecosystem, and across an organization to create a common set of practices for developers that help things both go more quickly and be more understandable and maintainable. This is a form of containing entropy. As we mature, we realize that picking one tool or framework or language or platform is not a matter of personal taste but rather a choice with broad ramifications for future flexibility, mergers and acquisitions, training, our ability to hire future supporting teams, and our future ability to directly support—or subvert—the business strategy.

Those with more business-oriented concerns and technologists cannot ignore each other's fields. Working as a patternmaker and a synthesizer, the architect-as-strategist broadens and ennobles these concerns, creating technology strategies that both are rooted in the causes and concerns of the business and recognize its constraints and opportunities. In collaboration with product management and with colleagues in strategy, business development, finance, and HR, the architect works to ensure there is alignment between the systems, yes, but also between those systems and the organization, and between the organization and its stated aims.

In short, for far too long we architects have thought we were in the business of making software. But we're in the business of building a business.

The architect who is containing entropy is stating a vision around which to rally, showing a path in a roadmap, garnering support for that vision through communication of guidelines and standards, and creating clarity to ensure efficiency of execution and that you're doing the right things and doing things right.

I love this definition of containing entropy because it offers something to both the software-minded and the business-minded architect (which I hope are two categories this book will help to collapse). One cannot be successful as an architect without thinking of not only what to do but how to get it done within an organization, which requires knowing why it should matter to someone who isn't a technologist.

We often hear of architects with failed dreams of how the system should have been. They are consumed by writing documents, and those documents are subsequently ignored, leading them to give up. Left with only the most informal conversational avenues to offer insufficient direction to teams, they become frustrated and even marginalized.

Knowing that you're in the business of building a business, and that technology is just an avenue by which you enable that, is a critical first step to being not only useful but powerful as an architect and strategist.

Specify nonfunctional requirements

Knowing what you're on the hook for, letting others know it, and making sure that it's a concrete deliverable will all go a long way to ensuring your vision is understood and realized.

Product management is responsible for specifying what the system must do for the end user. They might state functional requirements in user stories and epics.

The nonfunctional requirements are properties of the system that do not necessarily appear directly to the user. They are typically described as the "-ilities." The ones I focus on most are scalability, availability, maintainability, manageability, monitorability, extensibility, interoperability, portability, security, and performance.

The architect is responsible for specifying how the system will realize the functional and nonfunctional requirements in its construction. In order to do so, she must write a document that specifies how these will be realized.

This document, the architecture definition, serves as the technologist's answer to the blueprint. It should be structured in four broad categories to include business, application, data, and infrastructure perspectives and expressed with clarity and decisiveness, using primarily testable statements as valid propositions (which we'll examine in the next chapter) and math.

Finding ways to make those expressions concrete and executable is too often overlooked. In addition to writing and publishing a formal architecture definition document to the teams, you can do this by adding nonfunctional requirements to user stories as acceptance criteria.

Determine trade-offs

You can never try to escape one danger without encountering another. Prudence consists in recognizing the different dangers and in accepting the least bad as good.

—Machiavelli, The Art of War

As we know, every action produces an equal and opposite reaction. Adding security reduces performance. Sharding and partitioning the database affords greater performance and distribution but creates complexity that is difficult to manage. Adding robust monitoring can generate huge volumes of log data to be stored, rotated, secured, and cleansed. Keeping the design "simple" often defers the interests of flexibility until later, where it becomes very expensive.

The role of the architect is to see where those challenges may lurk, seek to make them explicit, and make value judgments about how to balance the solutions and the new problems they occasion, under the guidance of the broader business strategy. As English poet John Milton wrote in Paradise Lost, you make "the darkness visible."

In short, you're never quite solving a problem. You're only trading it for one that you'd rather have. We solve our need for shelter by assuming a mortgage for which we then must pay. Paul Virilio, the French cultural theorist and philosopher, reminds us lucidly, "When you invent the ship, you also invent the shipwreck...Every technology carries its own negativity, which is invented at the same time as technical progress" (Politics of the Very Worst, Semiotexte). Your architecture and strategy work will do well to examine not only how you are addressing the problems you've been given but also what new problems your solutions precipitate.

Any trade-off eventually reduces to a trade-off of time and money.

Absent a strategic mindset, many technologists left to their own devices create what amounts to little more than shopping lists of shiny objects. These can include the latest and most fashionable tech because it's popular or because it might bolster their résumé. We hear this frequently described as "a solution looking for a problem." Moreover, the less shallow or cynically minded among us are still rather prone to chasing exciting technology for its own sake, not unlike a dog chasing a squirrel. Intellectual curiosity is a wonderful thing, a best thing. But to ensure your technology and architecture decisions are truly supportive of the business—that is, give it the best chance to create competitive advantage—they need to be not shopping lists of shiny objects but squarely strategic.

So, let's look at the role of the strategist.

The Strategist's Role

Strategy is about getting more power than the starting position would suggest. Strategy is the art of creating power.

—Lawrence Freedman, Strategy: A History (Oxford University Press)

The word "strategy" originates from the Greek strategos. The term first appeared in fifth-century Athens as a conflation of the words meaning the expansion of the military general and came to be used to refer to the offices or science of the general—the general's work. But the word strategy entered general use only at the start of the 19th century in Antoine-Henri Jomini's writing on Napoleon's methods.

Jomini was of Swiss origin; he started out as a banker in Paris, later joined the French army under Napoleon, and eventually got promoted to general. Jomini began writing down Napoleon's methods in such a lucid manner that they came to be published as a book, entitled Treatise on Major Military Operations, in 1803. Jomini's strategies were employed in the U.S. Civil War and eventually taught at West Point Academy. He is considered the founder of modern strategy by many military historians.

Jomini's definition of strategy helpfully divides the word. He writes, "Strategy decides where to act; logistics brings the troops to this point; tactics decides the manner of execution." In other words, means (resources) are allocated and subjected to a method in order to achieve a goal.

Yet definitions of strategy vary. One of the more abstract definitions comes from Sun Tzu, a Chinese general and philosopher and author of The Art of War in 500 B.C. His book was not translated into English until the 20th century, at which point it began serving as a foundational text for guiding military strategies. It entered the popular imagination once it got adapted and marketed for business purposes.

In it, he writes, "Strategy is the art of making use of time and space." This is a tall order, and while aesthetically I appreciate the definition, we can break this down further in order to come to something practically executable.

Note The History of Strategy

If you're interested in the intellectual history of strategy, its origins, and its evolution from military thought to game theory to business, I highly recommend Lawrence Freedman's "Strategy: A History" (Oxford University Press). It's a fascinating read and offers a much richer view than we need here.

For our purposes, strategy is about determining the problems and opportunities in front of you, defining them properly, and shaping a course of action that will give your business the greatest advantage. Balancing problem solving with creating and exploiting new opportunities through imagination and analysis is the cornerstone of a great strategy.

Echoing Jomini, we'll say that strategy is about determining the best balance between a set of goals, the method used to achieve them, and the resources available as means. With the current rate of change in business, we can't set it and forget it, expecting that a three- or five-year strategy will go unrevised. At the same time, constant revision amounts to a reactionary collection of tactics, which is no strategy at all.

Most business strategies will concern themselves with the following:

  • The goals of the organization

  • The operating model: processes and how your company conducts its business

  • Culture: the mores and value system, the modes of communication

  • Talent strategy: how you source and retain talent, how you train them

  • Facilities strategy: where you do business, relevant local laws, and cost concerns

Strategies should be created at different levels: broad corporate-level strategies, business unit or division strategies, departmental strategies, and portfolio strategies. These will be more or less formal and be revised more frequently according to the climate and what life-cycle stage you find yourself in.

The Triumvirate: Strategy, Culture, and Execution

Culture eats strategy for breakfast.

—Management professor Peter Drucker

Any business aims to do one or many of these things:

  • Grow shareholder value

  • Grow earnings per share

  • Increase revenue

  • Manage costs

  • Diversify or create new revenue streams

  • Cross-sell more products

  • Increase market share

  • Increase share of wallet

  • Increase yield

  • Improve customer retention

  • Reduce product error/defect rates

  • Improve safety

  • Improve time to market/speed of operations

  • Grow through acquisition

Of course, there are different emphases at different times. To achieve these aims, broadly speaking, the strategist asks these questions:

  • Are resources devoted to the right areas, to the most important customers?

  • Are we creating products and services that can thrive in a market in different time horizons?

  • Where should we spend money? Where should we cut costs?

  • Where do skills need to be added or strengthened?

  • Where can productivity be improved?

  • What culture, attitude, and skills are required?

Many companies have a chief strategy officer or VP of corporate strategy. Strategy season frequently begins in the spring, giving this person and her team a couple of months to prepare a deck to present to the executive leadership team in the late summer. This will be discussed, revised, and eventually approved and used as input for budget season, which begins in the fall and continues until the budget for the following year is approved. We in technology tend to like to see our ideas realized moments after we have them. Being aware of this calendar and corporate planning process will help you plan for adding any big-ticket items to the slate in time for them to receive the necessary attention, support, and budget allocations.

That said, the evolution of Agile software methods, the preponderance of "disruptive" startups, and a growing global economy have all aligned variously to dilute the formality and rigidity of the strategist's role in such a process, leading her to rely more on regular conversations with the executive team and create reports with tighter scopes on an ongoing basis.

Depending on her level of power and position within the organization, the strategist finds herself concerned with some or all of the following:

  • Identifying business development opportunities, such as partnerships, joint ventures, cooperative arrangements with competitors, and the like

  • Finding, proposing, and validating mergers and acquisition opportunities

  • Building strategic capabilities within certain areas of the organization, such as helping create a sustainable AI practice in the face of growing trends

  • Performing research based on data to recommend long-term directions for the company (generally 1–3 years)

This last one is very common and how many strategists are trained as consultants entering the field at the venerable strategy firms such as Bain, Boston Consulting Group (BCG), and McKinsey. They likely work with business analysts, marketing, sales, technology, and operations teams in a cross-functional working group to develop hypotheses for how the business climate might be enabling or impinging upon their competitive advantage and how they should define a goal and direction and allocate resources to win in the marketplace.

According to one McKinsey report, 40% of strategists responding to their survey are most focused on "using fact-based analysis to spot industry shifts and to understand their own companies' sources of competitive advantage as a foundation for clear, differentiated strategies."

But spending months researching and creating data-driven decks is no longer enough. Because the world is moving so fast, the traditional strategist has taken the driver's seat in building capabilities. As the walls between business and technology continue to fold in on each other, the strategist may well find himself leading a team of data scientists to create an analytics platform to help themselves and their customers gain precious insights into their business operations. My colleague Balaji Krishnamurthy, the VP of strategy at Sabre Hospitality, who was previously in strategy roles at McKinsey and LinkedIn, offers this observation:

To be a good strategist, you need to be ready to deal with ambiguity. You need to be ready to pivot. You must form a hypothesis quickly about what must be done, then synthesize a lot of data. You must then see options and possibilities available, determine a goal, and present your findings clearly with a recommendation on how to allocate resources to achieve that goal.

Ultimately, companies are looking to grow and gain some distinctive competitive advantage. They can do this through technical innovation properly applied to real-world business problems. One assertion of this book is that the roles of chief architect and chief strategist are more blurred, and more aligned, than ever and that their mutual understanding of each other's concerns and methods will be an increasingly important driver for winning organizations.

Learn from your executive and product leadership teams what areas of focus they have for their business strategy and product roadmaps, so you can be prepared to match your technology to them.

For example, if your business is in cost-cutting mode, as companies tend to be when revenue is soft or they're preparing for an IPO, then your technology strategy should match. You can do that by examining the people angle: can you move workers or ramp down in expensive cities in order to hire programmers in lower-cost development centers? Can you examine your delivery and release processes to add automation and reduce manual labor there? Can you use free and open source libraries in place of expensive commercial software? These are examples from people, processes, and technical perspectives of how you can map your technology strategy to the business strategy.

A strategy deck is analogous to an architecture definition document for the organization. Neither will achieve the desired aims if you assume it lives in a vacuum.

The culture of your organization comprises your stated principles and to a far greater extent, the actual lived principles as reflected by the attitudes, communication styles, and behaviors of your teams. If your teams are territorial and competitive, an integrative platform strategy must identify and address that challenge.

Finally, your teams must be ready and capable of executing on your strategy. A strategy deck that states lofty, exciting aims will fail if it also doesn't include diligent, consistent execution and clear metrics to measure its success. This triumvirate is illustrated in Figure 1.

Figure 1. The triumvirate dominating forces of strategy, culture, and execution

Find ways to work with your leadership and across teams to ensure all of these forces are aligned. A good first step for doing so is to create two versions of the strategy: one that provides an honest and detailed examination of all three factors to share with the executive team and another shorter version that communicates only the changes you're driving in a way that you can share publicly with teams. In long-range planning, there are financial, business transaction, and personnel matters that obviously can't be disclosed.

Continue reading How architecture evolves into strategy.

Categories: Technology

Four short links: 22 October 2018

O'Reilly Radar - Mon, 2018/10/22 - 03:55

Perl in the Browser, Pharo Programming, Program Synthesis, and Raster Vision

  1. WebPerl -- run Perl in the browser, via WebAssembly and EmScripten. PerlMonks discussion. (via Hacker News)
  2. Pharo -- a pure object-oriented programming language and a powerful environment focused on simplicity and immediate feedback (think IDE and OS rolled into one). SmallTalk's ideas are ready for a comeback!
  3. Type-Driven Program Synthesis -- The talk will present two applications of type-driven synthesis. The first one is a tool called Synquid, which creates recursive functional programs from scratch given a refinement type as input. Synquid is the first synthesizer powerful enough to automatically discover programs that manipulate complex data structures, such as balanced trees and propositional formulas. The second application is a language called Lifty, which uses type-driven synthesis to repair information flow leaks. In Lifty, the programmer specifies expressive information flow policies by annotating the sources of sensitive data with refinement types, and the compiler automatically inserts access checks necessary to enforce these policies across the code.
  4. Raster Vision -- open source framework for deep learning on satellite and aerial imagery.

Continue reading Four short links: 22 October 2018.

Categories: Technology

10 top Python resources on O’Reilly’s online learning platform

O'Reilly Radar - Mon, 2018/10/22 - 03:00

Our most-used Python resources will help you stay on track in your journey to learn and apply Python.

We dove into the data on our online learning platform to identify the most-used Python resources. These are the items our platform subscribers regularly turn to as they apply Python in their projects and organizations.

Introduction to Python — Jessica McKellar teaches you Python’s core concepts and data types through hands-on exercises and useful projects.

Python for Data Analysis, 2nd Edition — Wes McKinney shares practical case studies that show you how to solve a broad set of data analysis problems effectively. You’ll learn the latest versions of pandas, NumPy, IPython, and Jupyter in the process.

Learning Python, 5th Edition — Mark Lutz leads you through an in-depth, hands-on introduction to the core Python language. It’s an ideal way to begin, whether you’re new to programming or a professional developer versed in other languages.

Deep Learning with Python — François Chollet introduces the field of deep learning using the Python language and the powerful Keras library. This book builds your understanding through intuitive explanations and practical examples.

Learn Python 3 the Hard Way: A Very Simple Introduction to the Terrifyingly Beautiful World of Computers and Code, 1st Edition — Zed A. Shaw helps you learn Python using 52 exercises. Read them. Type their code precisely. Fix your mistakes. Watch the programs run.

Fluent Python — Luciano Ramalho takes you through Python’s core language features and libraries, and shows you how to make your code shorter, faster, and more readable at the same time.

Automate the Boring Stuff with Python — Al Sweigart teaches simple programming skills to automate everyday computer tasks.

Python Crash Course — Eric Matthes takes you through a fast-paced, thorough introduction to programming with Python that will have you writing programs, solving problems, and making things that work in no time.

Effective Python: 59 Specific Ways to Write Better Python — Brett Slatkin helps you master a truly “Pythonic” approach to programming, bringing together 59 Python best practices, tips, and shortcuts, and explaining them with realistic code examples.

Python Cookbook, 3rd Edition — Packed with practical recipes written and tested with Python 3.3, this unique cookbook by David Beazley and Brian K. Jones is for experienced Python programmers who want to focus on modern tools and idioms.

Continue reading 10 top Python resources on O’Reilly’s online learning platform.

Categories: Technology

Four short links: 19 October 2018

O'Reilly Radar - Fri, 2018/10/19 - 03:55

PDF to Data Frame, Clever Story, Conceptual Art, and Automatic Patch Synthesis

  1. Camelot -- Python library that extracts tables of data from PDF documents, returning them as Pandas frames.
  2. STET -- short story told via footnotes, editorial markup, and more. Magnificent! (via Cory Doctorow)
  3. Solving Sol -- interpreting a conceptual artist's art as instructions, reframed as an AI problem. Clever!
  4. Human-Competitive Patches with Repairnator -- Repairnator is a bot. It constantly monitors software bugs discovered during continuous integration of open source software and tries to fix them automatically. If it succeeds to synthesize a valid patch, Repairnator proposes the patch to the human developers, disguised under a fake human identity. To date, Repairnator has been able to produce five patches that were accepted by the human developers and permanently merged in the code base.

Continue reading Four short links: 19 October 2018.

Categories: Technology

Four short links: 18 October 2018

O'Reilly Radar - Thu, 2018/10/18 - 03:50

Git Playbook, Lessons Learned, Neural NLP, and Landscape Generation

  1. Flight Rules for Git -- the hard-earned body of knowledge recorded in manuals that list, step-by-step, what to do if X occurs and why. Essentially, they are extremely detailed, scenario-specific standard operating procedures. What to do after you shoot yourself in the foot in interesting ways with Git.
  2. Lessons Learned from Creating a Rich-Text Editor with Real-Time Collaboration -- This article describes how we approached the problem and what challenges we had to overcome in order to provide real-time collaborative editing capable of handling rich text. Check it out if you are interested in: learning what problems you may face when implementing real-time collaborative editing, building a rich-text editor with support for real-time collaboration, and how we approached collaborative editing in CKEditor 5.
  3. A Review of the Recent History of Natural Language Processing -- This post will discuss major recent advances in NLP focusing on neural network-based methods.
  4. Landscape -- software that builds the Cloud-Native Computing Foundation's landscape of products.

Continue reading Four short links: 18 October 2018.

Categories: Technology

What operations professionals need to know to fuel career advancement

O'Reilly Radar - Thu, 2018/10/18 - 02:10

O’Reilly’s new survey reveals the latest operations salary trends, and the skill sets that will keep your operations career on track.

O’Reilly conducted a recent survey[1] of operations professionals, and the results offer useful information and insights to empower your career planning. As you’d expect, the survey revealed that respondents put emphasis on their salaries when evaluating their careers, but they also pay close attention to company and team attributes, job activities, role responsibilities, and evolving skill set requirements.

How operations salaries add up

Survey results show that in 2018, the median annual salary for operations professionals clocks in at $90,000. Salary increases with age and experience: someone with more than 20 years of experience can earn a median income of around $123,000.

Figure 1. Operations salaries by years of experience. Image credit: O'Reilly. The company, team, and industry all make a difference

The larger the company, the more you should expect to earn. For example, the median salary for companies employing two-to-100 people is slightly more than $78,000. Jump to companies with more than 10,000 employees and the average income rises to $114,000. Interestingly, the age of a company is not a huge factor in determining compensation.

Team size, however, does make a difference among survey respondents. The general trend is that the larger the team size, the higher the median salary. Keep in mind that joining a bigger team does not necessarily equate to a pay increase. Larger teams usually mean more senior team members, team leads, and an established hierarchy. Increased responsibility generates increased compensation.

The industry where you work does affect compensation. About a third of survey respondents work in the software industry, and they report a median salary of $95,000. Operations professionals working for high-paying health care and medical companies see a median salary of $113,000.

Where time spent impacts dollars earned

It seems the more coding you do as part of your job, the less you earn. For survey respondents who code one-to-three hours per week, the median salary is around $94,000. Spend 20 hours or more per week on code tasks and the median salary drops to $82,000. You can attribute this to several factors. One, as you become more senior in your organization, increased responsibilities leave less time for coding. And two, if you are part of an organization with many coders, both entry-level staff and interns bring down the median salary.

For those not fond of attending meetings, here’s a survey result you might not want to see: the more time you spend in meetings, the higher the median salary. Those who spend more than 20 hours per week in meetings have a median salary of $140,000. Of course, meetings can be a proxy for responsibility, so booking yourself into every optional meeting will not increase salary automatically.

Speaking the same programming language

Scripting languages are the most popular programming languages among respondents, with Bash being the most used (66% of respondents), followed by Python (63%), and JavaScript (42%).

Go is used by 20% of respondents, and those who use Go tend to have one of the higher median salaries at $102,000, similar to LISP and Swift. This could be related to the types of companies that are pushing these programming languages. Google and Apple, for example, are very large companies and, as noted, salary and company size are related.

And what about the operating system in which respondents work? Linux tops the charts at 87% usage. Windows is also used frequently (63%), often as a mix between workstations and servers, and in some cases as a front end for Linux/Unix servers.

Education pays

Computer science, mathematics, statistics, and physics are the top fields of study for operations professionals. Advanced degrees do have a positive impact on salary. The median salary among respondents for those with a master’s is $82,000, whereas a doctorate garners a median salary of $98,000.

Planning your next operations career move

One third of survey respondents agree that the next best step to career advancement is to learn a new skill or technology. This makes sense, as the technology landscape is evolving quickly and you need to acquire new skills to keep up.

Wanting to work on more interesting or important projects is a motivator for career change among some respondents (25%), as is the desire to move into leadership roles (15%). Only 12% of respondents want to switch companies.

Other things respondents keep top of mind when pondering their operations career paths include non-monetary compensation such as job flexibility, work-life balance, location, and company culture.

Looking for more data to guide your career development? Download the 2018 Annual IT/Ops Salary Survey for free.

[1] Operations professionals answered a range of questions about their current roles. More than 1,300 respondents from 70-plus countries participated in the survey.

Continue reading What operations professionals need to know to fuel career advancement.

Categories: Technology

Four short links: 17 October 2018

O'Reilly Radar - Wed, 2018/10/17 - 04:15

Reservoir Computing, ProxyJump, SID Sequencer, and 2KB AI

  1. MEMS Neuromorphic Computing -- the construction of the first reservoir computing device built with a microelectromechanical system (MEMS). [...] [T]he neural network exploits the nonlinear dynamics of a microscale silicon beam to perform its calculations. The group's work looks to create devices that can act simultaneously as a sensor and a computer using a fraction of the energy a normal computer would use. Early-stage research but an interesting direction for the future of hardware.
  2. SSH ProxyJump -- it’s somewhat common to have what’s known as a “jump host” serve as an SSH gateway to a remote network. You use SSH to log into the jump host (or “jump server”) and from there use SSH to log into an internal host that’s not directly accessible from the internet. This useful utility makes it a one-step action.
  3. Booting defMON -- an introduction to an absolutely wild low-level sequencer for the C64 SID chips.
  4. Machine Learning on 2KB of RAM -- This paper develops a novel tree-based algorithm, called Bonsai, for efficient prediction on IoT devices—such as those based on the Arduino Uno board having an 8-bit ATmega328P microcontroller operating at 16 MHz with no native floating point support, 2KB RAM, and 32KB read-only flash. (jaws drop)

Continue reading Four short links: 17 October 2018.

Categories: Technology

Four short links: 16 October 2018

O'Reilly Radar - Tue, 2018/10/16 - 04:05

Common Sense, Photorealistic Rendering, Logic Game, and the Grey-hat Patcher

  1. Teaching Machines Common Sense Reasoning (DARPA) -- To focus this new effort, MCS will pursue two approaches for developing and evaluating different machine common sense services. The first approach will create computational models that learn from experience and mimic the core domains of cognition as defined by developmental psychology. [...] The second MCS approach will construct a common sense knowledge repository capable of answering natural language and image-based queries about common sense phenomena by reading from the web.
  2. Physically Based Rendering -- a textbook that describes both the mathematical theory behind a modern photorealistic rendering system as well as its practical implementation.
  3. QED -- a short interactive text in propositional logic arranged in the format of a computer game.
  4. A Mysterious Grey-Hat is Patching MicroTik Routers -- "I added firewall rules that blocked access to the router from outside the local network," Alexey said. "In the comments, I wrote information about the vulnerability and left the address of the @router_os Telegram channel, where it was possible for them to ask questions." More helpful than some corporate IT departments...

Continue reading Four short links: 16 October 2018.

Categories: Technology

4 imperatives for making business intelligence work

O'Reilly Radar - Tue, 2018/10/16 - 03:50

Create a coherent BI strategy that aligns data collection and analytics with the general business strategy.

Results-based leaders rely on having the right information at the right time in order to support operational decision-making. That’s why decision-makers consider business intelligence their top technology priority. They recognize the instrumental role data plays in creating value and see information as the lifeblood of the organization. They then use actionable insights to confidently and consistently lead by delivering results that count.

The business intelligence (BI) and data science industries have spent the last couple decades making data access easier, analytic capability more comprehensive, and platforms more scalable. Yet, despite pouring billions of dollars into BI initiatives, executives often come up empty-handed when they reach for the information they need to make well-informed decisions. Executives fail to fully capitalize on BI’s promise of turning actionable insights into real business value when BI efforts aren’t planned or executed effectively. These problems are further compounded as companies move to adopt more sophisticated data science and AI. To achieve the results that leaders are looking for, organizations must create a coherent BI strategy that aligns data collection and analytics with the general business strategy.

Our experience shows that by focusing on four actionable steps, or imperatives, we can empower business leaders to adequately address planning and execution challenges to build a decision support competency that works.

Step 1: Unify

What we believe influences how we behave, and unifying your organization begins with aligning many unique and often divergent perspectives across different business divisions on business intelligence and analytics. Senior leaders across an organization must collaborate efficiently for BI to be successful.

All too often, requests for information from the business go unanswered, as different siloed departments trip over themselves to coordinate inter-departmental cooperation. Technical nuances around data and data wrangling are often misunderstood and miscommunicated because practitioners routinely fail to understand key business requirements. Business leaders need to look for data science candidates with keen technical, analytic, and business acumen (full disclosure: Michael Li is the founder and CEO of The Data Incubator) to unify their BI efforts between technical and non-technical parts of the business.

Business intelligence is a business initiative, not a tech project. It’s an ongoing effort across an entire organization to improve its decision-making ability to create and maximize value. There is no finish line. Adopting this attitude across every business division in your organization is a prerequisite for effective collaboration and a necessity for creating the kind of cross-functional alignment needed for BI success.

Step 2: Simplify

Complexity is wreaking havoc on businesses and making it increasingly difficult for decision-makers to create value. Analytics works best when the process of moving from great idea to actionable insight is fast, focused, and uncomplicated.

To simplify your BI efforts, start by building key alliances with critical stakeholders in different lines of business within your organization. Now more than ever, CEOs rely on CIOs and CDOs to drive an organization’s value-creation agenda, and that makes effective collaboration between business and IT absolutely critical to BI success. The days of an ivory-towered BI detached from real business operations are over. It is vital that business leaders work overtime to bridge the all-too-common communication, trust, and understanding gaps.

Then, secure executive buy-in and the financial resources you need for your efforts by building your capability one incremental step at a time and demonstrating real value every step of the way. When building your BI capability, always start with the existing technology you already have. Most organizations have already made significant investments in tools and infrastructure and have built important intellectual capital that only comes with experience and time. Prove that it can’t or won’t work before requesting additional funds for new technology.

Finally, when it comes to providing decision-makers with the information they need to do their jobs, minimizing time-to-results is critical. This means striking the right balance between governing and enabling the business to perform without hindering innovation and creativity.

Step 3: Amplify

Skeptics and naysayers exist in every organization. They prefer the status quo, resist change, and make comments like, “we’ve been down this road before,” and “I’ll believe it when I see it.” At best, they’re stubborn “demanders of proof,” willing to believe only when presented with concrete results. At worst, they’re obstructionists—preventing business intelligence initiatives from realizing their full potential.

As BI evolves from traditional reporting and descriptive analytics toward data science and AI, many practitioners fear that new capabilities will make their skill sets obsolete.Fighting new initiatives is, perhaps, a natural preservation instinct. The prevalence of naysayers may also be symptomatic of cultural biases in the institution. Deloitte refers to it as the “inertia of good intentions”—personal behavior created by institutional routines, obligations, and pressures that actually hold many back (unsuspectingly) from delivering the kind of value their organizations need. Left unattended, the culture of most organizations can marginalize BI initiatives to the point of limited and unacceptable return.

You can avoid the negative impact of skeptics and naysayers as well as a culture of mistrust by establishing organizational awareness and building excitement around BI, analytics, and data science initiatives. To amplify means to evangelize.

For instance, large enterprises often create a Data Science Competency Center or AI Center of Excellence, which helps lead the effort to modernize analytics. These evangelists define the data science and AI practices for the firm and are responsible for elevating the general analytical skill level of the entire organization. Fortune 500 Data Science Centers of Excellence are hosting in-depth trainings in data and AI to help bridge the skills gap between the advanced data science practitioners of their organizations and the typical rank-and-file analysts.

Step 4: Qualify

Business intelligence is a journey—a process of continuous improvement meant to adapt and evolve so that business leaders can give intelligent responses to an ever-changing and dynamic business environment. After all, what decision-makers need to monitor and evaluate the business today will change tomorrow. The only way for a business to keep pace is for its reporting and analytics capabilities to keep pace as well.

Today, few firms qualify success properly. They don’t proactively monitor and measure BI performance against end-user expectations and real business outcomes, so they can’t effectively evolve.

Ensure that you focus adequate attention on active monitoring, evaluation, and adjustment of your organization’s business intelligence capabilities so they’re always aligned with the business’ needs and always responsive to stakeholder expectations.

As companies are looking toward growing their BI, analytics, and data science departments, management is demanding results. All too often, analytics projects fall short because leaders fail to understand the key elements of a successful analytics strategy while creating one. In order to plan and execute successful business intelligence efforts, leaders in this area must adopt these imperatives. By focusing your organization’s BI initiative around simplifying, unifying, amplifying, and qualifying business intelligence within the whole organization, you’ll be able to make smarter business decisions, deliver successful results, and keep your firm ahead of the competition.

Continue reading 4 imperatives for making business intelligence work.

Categories: Technology

Four short links: 15 October 2018

O'Reilly Radar - Mon, 2018/10/15 - 03:55

Robots, Cryptocurrencies, Bayes, and Brains

  1. What People See in a Robot (YouTube) -- In a study using 24 robots selected from this three-dimensional appearance space, I then show that the different dimensions separately predict inferences people make about the robot’s affective, social-moral, and physical capacities. (via RoboHub)
  2. Crypto is the Mother of All Scams and (Now Busted) Bubbles While Blockchain Is The Most Over-Hyped Technology Ever, No Better than a Spreadsheet/Database (Nouriel Roubini) -- Roubini's testimony to the Hearing of the U.S. Senate Committee on Banking, Housing and Community Affairs on Blockchains. It is clear by now that Bitcoin and other cryptocurrencies represent the mother of all bubbles, which explains why literally every human being I met between Thanksgiving and Christmas of 2017 asked me first if they should buy them. [...] A chart of Bitcoin prices compared to other famous historical bubbles and scams—like Tulip-mania, the Mississippi Bubble, the South Sea Bubble—shows that the price increase of Bitcoin and other crypto junkcoins was 2X or 3X bigger than previous bubbles, and the ensuing collapse and bust as fast and furious and deeper. [...] Actually calling this useless vaporware garbage a “shitcoin” is a grave insult to manure that is a most useful, precious, and productive good as a fertilizer in agriculture. It's all quotable. Read it.
  3. Bayes' Theorem in the 21st Century -- I recently completed my term as editor of an applied statistics journal. Maybe a quarter of the papers used Bayes’ theorem. Almost all of these were based on uninformative priors, reflecting the fact that most cutting-edge science does not enjoy Five-Thirty-Eight-level background information. Are we in for another Bayesian bust?
  4. Numenta's New Theory -- research paper, talk, NYT story. Will be interesting to see how this fares in peer review.

Continue reading Four short links: 15 October 2018.

Categories: Technology

Four short links: 12 October 2018

O'Reilly Radar - Fri, 2018/10/12 - 03:55

Activity Alert, JavaScript Visualizations, OT vs. CRDT, and Senior Engineering

  1. Publicly Available Tools Seen in Cyber Incidents Worldwide (US-CERT) -- The tools detailed in this activity alert fall into five categories: remote access trojans (RATs), webshells, credential stealers, lateral movement frameworks, and command and control (C2) obfuscators. This activity alert provides an overview of the threat posed by each tool, along with insight into where and when it has been deployed by threat actors. Measures to aid detection and limit the effectiveness of each tool are also described. The activity alert concludes with general advice for improving network defense practices.
  2. Muze -- Tableau-like visualizations in JavaScript. Open source (MIT).
  3. Real Differences between OT and CRDT for Co-Editors -- key CRDT design issues include designing CRDT-special data structures and schemes for representing and manipulating object sequences, searching and executing identifier-based operations in the object sequence, and conversions between internal identifier-based operations and external position-based operations, which collectively deal with both application-specific and concurrency issues in co-editing. This approach has induced a myriad of CRDT-specific challenges and puzzles, such as the correctness of key CRDT data structures and functional components, tombstone overhead, variable and lengthy identifiers, inconsistent-position-integer-ordering and infinite loop flaws, position-order-violation puzzles, and concurrent-insert-interleaving puzzles.
  4. What's a Senior Engineer's Job? (Julia Evans) -- I want to talk here about the work that a senior engineer does.

Continue reading Four short links: 12 October 2018.

Categories: Technology

Learn about serverless with these books, videos, and tutorials

O'Reilly Radar - Fri, 2018/10/12 - 03:00

This collection of serverless resources will get you up to speed on the basics and best practices.

Whether you’re just getting started with serverless or you have previous experience, you’ll find something useful on this list of serverless resources.

The items on this list were curated by O’Reilly’s editorial experts.

Getting started with serverless

Use this introductory material to get up to speed on the basics of serverless.

An Introduction to Serverless — In this short overview, Mike Roberts introduces the concepts behind serverless architectures.

What is Serverless? — Mike Roberts and John Chapin take you through the serverless landscape—particularly the design considerations, tooling, and approaches to operational management you need to make it work.

Learning Path: Getting Started with Serverless — Sam Newman demonstrates the benefits of serverless and provides overviews of function as a service (FaaS) and back end as a service (BaaS).

Implementing serverless

These resources outline best practices for incorporating serverless into your organization.

Learning Path: Migrating Microservices to Serverless — Sam Newman guides you through the many serverless frameworks that are currently available so you can choose the one that’s best for your organization.

Serverless Ops — Michael Hausenblas explores several use cases where serverless is a great fit—primarily short-running, stateless jobs in event-driven architectures found in mobile or IoT applications.

Going Serverless: Security Outside the Box — Jack Naglieri and Austin Byers explore tools and techniques for successfully building, deploying, and debugging serverless security applications.

Lessons Learned from Operating a Serverless-like Platform at Scale — Sangeeta Narayanan shares insights from operating Netflix’s customizable API, which allows the creation of optimized experiences on 1,000+ devices through a serverless-like platform and experience.

Building and Running Serverless Data Pipelines on AWS — Mike Roberts walks through a real-life example of a platform that was rearchitected to provide increased data capacity, reduced cost, and an improved development cycle time.

Microservice Orchestration for Serverless Computing — Cathy Zhang explains how service graphs address the challenge of creating and managing microservice applications.

Continue reading Learn about serverless with these books, videos, and tutorials.

Categories: Technology

Bringing AI into the enterprise: A functional approach to the technologies of intelligence

O'Reilly Radar - Thu, 2018/10/11 - 13:00

Kristian Hammond maps out simple rules, useful metrics, and where AI should live in the org chart.

Continue reading Bringing AI into the enterprise: A functional approach to the technologies of intelligence.

Categories: Technology

Building artificial people: Endless possibilities and the dark side

O'Reilly Radar - Thu, 2018/10/11 - 13:00

Supasorn Suwajanakorn discusses the possibilities and the dark side of building artificial people.

Continue reading Building artificial people: Endless possibilities and the dark side.

Categories: Technology

Four short links: 11 October 2018

O'Reilly Radar - Thu, 2018/10/11 - 10:20

Decentralized Applications, Global Startups, Better Shuffling, and Prolog Text

  1. Decentralized Applications (MIT) -- interesting course to be taught by Robert T Morris. The goal of 6.S974 is to understand recent efforts in decentralized applications, to learn what the main design trade-offs are, and to identify areas for new research. My spidey-sense is tingling. This has all the hallmarks of one of those courses whose graduates build the next wave of companies and research areas.
  2. America Is Losing Its Startup Edge -- ignore the use of percentages and Decline of Roman^W American Empire alarmism, it's the rise of the rest of the world that's fascinating here. While it is true that venture-capital investment in the U.S. continues to rise, having reached more than $90 billion in 2017, such investment is growing even faster in other parts of the world, expanding by nearly 375%—more than twice the 160% increase here. China saw the largest jump, its share expanding from 4% of global venture investment in 2005 to a nearly a quarter of it by 2017.
  3. Playlist Shuffle -- This paper proposes a novel approach at shuffling a looping sequence that minimizes caveats of naive solutions, keeps computation low, and offers a high degree of variance. [...] The problem is how to repeatedly shuffle a cyclic list and avoid too close and too far duplicates.
  4. Art of Prolog, 2E -- this 1994 classic is now an open access title, free PDF download. Prolog is rational AI magic, while deep learning is intuitive AI magic.

Continue reading Four short links: 11 October 2018.

Categories: Technology

Fireside chat with Marc Warner and Louis Barson

O'Reilly Radar - Thu, 2018/10/11 - 10:00

Marc Warner and Louis Barson discuss the internal and external uses of AI in the UK government.

Continue reading Fireside chat with Marc Warner and Louis Barson.

Categories: Technology

Deep learning at scale: A field manual

O'Reilly Radar - Thu, 2018/10/11 - 10:00

Jason Knight offers an overview of the state of the field for scaling training and inference across distributed systems.

Continue reading Deep learning at scale: A field manual.

Categories: Technology

The missing piece

O'Reilly Radar - Thu, 2018/10/11 - 10:00

Cassie Kozyrkov shares machine learning lessons learned at Google and explains what they mean for applied data science.

Continue reading The missing piece.

Categories: Technology


Subscribe to LuftHans aggregator