Feed aggregator

PDF to Data Frame, Clever Story, Conceptual Art, and Automatic Patch Synthesis

1. Camelot -- Python library that extracts tables of data from PDF documents, returning them as Pandas frames.
2. STET -- short story told via footnotes, editorial markup, and more. Magnificent! (via Cory Doctorow)
3. Solving Sol -- interpreting a conceptual artist's art as instructions, reframed as an AI problem. Clever!
4. Human-Competitive Patches with Repairnator -- Repairnator is a bot. It constantly monitors software bugs discovered during continuous integration of open source software and tries to fix them automatically. If it succeeds to synthesize a valid patch, Repairnator proposes the patch to the human developers, disguised under a fake human identity. To date, Repairnator has been able to produce five patches that were accepted by the human developers and permanently merged in the code base.

Continue reading Four short links: 19 October 2018.

Git Playbook, Lessons Learned, Neural NLP, and Landscape Generation

1. Flight Rules for Git -- the hard-earned body of knowledge recorded in manuals that list, step-by-step, what to do if X occurs and why. Essentially, they are extremely detailed, scenario-specific standard operating procedures. What to do after you shoot yourself in the foot in interesting ways with Git.
2. Lessons Learned from Creating a Rich-Text Editor with Real-Time Collaboration -- This article describes how we approached the problem and what challenges we had to overcome in order to provide real-time collaborative editing capable of handling rich text. Check it out if you are interested in: learning what problems you may face when implementing real-time collaborative editing, building a rich-text editor with support for real-time collaboration, and how we approached collaborative editing in CKEditor 5.
3. A Review of the Recent History of Natural Language Processing -- This post will discuss major recent advances in NLP focusing on neural network-based methods.
4. Landscape -- software that builds the Cloud-Native Computing Foundation's landscape of products.

Continue reading Four short links: 18 October 2018.

O’Reilly’s new survey reveals the latest operations salary trends, and the skill sets that will keep your operations career on track.

O’Reilly conducted a recent survey[1] of operations professionals, and the results offer useful information and insights to empower your career planning. As you’d expect, the survey revealed that respondents put emphasis on their salaries when evaluating their careers, but they also pay close attention to company and team attributes, job activities, role responsibilities, and evolving skill set requirements.

How operations salaries add up

Survey results show that in 2018, the median annual salary for operations professionals clocks in at $90,000. Salary increases with age and experience: someone with more than 20 years of experience can earn a median income of around $123,000.

Figure 1. Operations salaries by years of experience. Image credit: O'Reilly. The company, team, and industry all make a difference

The larger the company, the more you should expect to earn. For example, the median salary for companies employing two-to-100 people is slightly more than $78,000. Jump to companies with more than 10,000 employees and the average income rises to $114,000. Interestingly, the age of a company is not a huge factor in determining compensation.

Team size, however, does make a difference among survey respondents. The general trend is that the larger the team size, the higher the median salary. Keep in mind that joining a bigger team does not necessarily equate to a pay increase. Larger teams usually mean more senior team members, team leads, and an established hierarchy. Increased responsibility generates increased compensation.

The industry where you work does affect compensation. About a third of survey respondents work in the software industry, and they report a median salary of $95,000. Operations professionals working for high-paying health care and medical companies see a median salary of $113,000.

Where time spent impacts dollars earned

It seems the more coding you do as part of your job, the less you earn. For survey respondents who code one-to-three hours per week, the median salary is around $94,000. Spend 20 hours or more per week on code tasks and the median salary drops to $82,000. You can attribute this to several factors. One, as you become more senior in your organization, increased responsibilities leave less time for coding. And two, if you are part of an organization with many coders, both entry-level staff and interns bring down the median salary.

For those not fond of attending meetings, here’s a survey result you might not want to see: the more time you spend in meetings, the higher the median salary. Those who spend more than 20 hours per week in meetings have a median salary of $140,000. Of course, meetings can be a proxy for responsibility, so booking yourself into every optional meeting will not increase salary automatically.

Speaking the same programming language

Scripting languages are the most popular programming languages among respondents, with Bash being the most used (66% of respondents), followed by Python (63%), and JavaScript (42%).

Go is used by 20% of respondents, and those who use Go tend to have one of the higher median salaries at $102,000, similar to LISP and Swift. This could be related to the types of companies that are pushing these programming languages. Google and Apple, for example, are very large companies and, as noted, salary and company size are related.

And what about the operating system in which respondents work? Linux tops the charts at 87% usage. Windows is also used frequently (63%), often as a mix between workstations and servers, and in some cases as a front end for Linux/Unix servers.

Education pays

Computer science, mathematics, statistics, and physics are the top fields of study for operations professionals. Advanced degrees do have a positive impact on salary. The median salary among respondents for those with a master’s is $82,000, whereas a doctorate garners a median salary of $98,000.

Planning your next operations career move

One third of survey respondents agree that the next best step to career advancement is to learn a new skill or technology. This makes sense, as the technology landscape is evolving quickly and you need to acquire new skills to keep up.

Wanting to work on more interesting or important projects is a motivator for career change among some respondents (25%), as is the desire to move into leadership roles (15%). Only 12% of respondents want to switch companies.

Other things respondents keep top of mind when pondering their operations career paths include non-monetary compensation such as job flexibility, work-life balance, location, and company culture.

Looking for more data to guide your career development? Download the 2018 Annual IT/Ops Salary Survey for free.

[1] Operations professionals answered a range of questions about their current roles. More than 1,300 respondents from 70-plus countries participated in the survey.

Continue reading What operations professionals need to know to fuel career advancement.

Reservoir Computing, ProxyJump, SID Sequencer, and 2KB AI

1. MEMS Neuromorphic Computing -- the construction of the first reservoir computing device built with a microelectromechanical system (MEMS). [...] [T]he neural network exploits the nonlinear dynamics of a microscale silicon beam to perform its calculations. The group's work looks to create devices that can act simultaneously as a sensor and a computer using a fraction of the energy a normal computer would use. Early-stage research but an interesting direction for the future of hardware.
2. SSH ProxyJump -- it’s somewhat common to have what’s known as a “jump host” serve as an SSH gateway to a remote network. You use SSH to log into the jump host (or “jump server”) and from there use SSH to log into an internal host that’s not directly accessible from the internet. This useful utility makes it a one-step action.
3. Booting defMON -- an introduction to an absolutely wild low-level sequencer for the C64 SID chips.
4. Machine Learning on 2KB of RAM -- This paper develops a novel tree-based algorithm, called Bonsai, for efficient prediction on IoT devices—such as those based on the Arduino Uno board having an 8-bit ATmega328P microcontroller operating at 16 MHz with no native floating point support, 2KB RAM, and 32KB read-only flash. (jaws drop)

Continue reading Four short links: 17 October 2018.

Common Sense, Photorealistic Rendering, Logic Game, and the Grey-hat Patcher

1. Teaching Machines Common Sense Reasoning (DARPA) -- To focus this new effort, MCS will pursue two approaches for developing and evaluating different machine common sense services. The first approach will create computational models that learn from experience and mimic the core domains of cognition as defined by developmental psychology. [...] The second MCS approach will construct a common sense knowledge repository capable of answering natural language and image-based queries about common sense phenomena by reading from the web.
2. Physically Based Rendering -- a textbook that describes both the mathematical theory behind a modern photorealistic rendering system as well as its practical implementation.
3. QED -- a short interactive text in propositional logic arranged in the format of a computer game.
4. A Mysterious Grey-Hat is Patching MicroTik Routers -- "I added firewall rules that blocked access to the router from outside the local network," Alexey said. "In the comments, I wrote information about the vulnerability and left the address of the @router_os Telegram channel, where it was possible for them to ask questions." More helpful than some corporate IT departments...

Continue reading Four short links: 16 October 2018.

Create a coherent BI strategy that aligns data collection and analytics with the general business strategy.

Results-based leaders rely on having the right information at the right time in order to support operational decision-making. That’s why decision-makers consider business intelligence their top technology priority. They recognize the instrumental role data plays in creating value and see information as the lifeblood of the organization. They then use actionable insights to confidently and consistently lead by delivering results that count.

The business intelligence (BI) and data science industries have spent the last couple decades making data access easier, analytic capability more comprehensive, and platforms more scalable. Yet, despite pouring billions of dollars into BI initiatives, executives often come up empty-handed when they reach for the information they need to make well-informed decisions. Executives fail to fully capitalize on BI’s promise of turning actionable insights into real business value when BI efforts aren’t planned or executed effectively. These problems are further compounded as companies move to adopt more sophisticated data science and AI. To achieve the results that leaders are looking for, organizations must create a coherent BI strategy that aligns data collection and analytics with the general business strategy.

Our experience shows that by focusing on four actionable steps, or imperatives, we can empower business leaders to adequately address planning and execution challenges to build a decision support competency that works.

Step 1: Unify

What we believe influences how we behave, and unifying your organization begins with aligning many unique and often divergent perspectives across different business divisions on business intelligence and analytics. Senior leaders across an organization must collaborate efficiently for BI to be successful.

All too often, requests for information from the business go unanswered, as different siloed departments trip over themselves to coordinate inter-departmental cooperation. Technical nuances around data and data wrangling are often misunderstood and miscommunicated because practitioners routinely fail to understand key business requirements. Business leaders need to look for data science candidates with keen technical, analytic, and business acumen (full disclosure: Michael Li is the founder and CEO of The Data Incubator) to unify their BI efforts between technical and non-technical parts of the business.

Business intelligence is a business initiative, not a tech project. It’s an ongoing effort across an entire organization to improve its decision-making ability to create and maximize value. There is no finish line. Adopting this attitude across every business division in your organization is a prerequisite for effective collaboration and a necessity for creating the kind of cross-functional alignment needed for BI success.

Step 2: Simplify

Complexity is wreaking havoc on businesses and making it increasingly difficult for decision-makers to create value. Analytics works best when the process of moving from great idea to actionable insight is fast, focused, and uncomplicated.

To simplify your BI efforts, start by building key alliances with critical stakeholders in different lines of business within your organization. Now more than ever, CEOs rely on CIOs and CDOs to drive an organization’s value-creation agenda, and that makes effective collaboration between business and IT absolutely critical to BI success. The days of an ivory-towered BI detached from real business operations are over. It is vital that business leaders work overtime to bridge the all-too-common communication, trust, and understanding gaps.

Then, secure executive buy-in and the financial resources you need for your efforts by building your capability one incremental step at a time and demonstrating real value every step of the way. When building your BI capability, always start with the existing technology you already have. Most organizations have already made significant investments in tools and infrastructure and have built important intellectual capital that only comes with experience and time. Prove that it can’t or won’t work before requesting additional funds for new technology.

Finally, when it comes to providing decision-makers with the information they need to do their jobs, minimizing time-to-results is critical. This means striking the right balance between governing and enabling the business to perform without hindering innovation and creativity.

Step 3: Amplify

Skeptics and naysayers exist in every organization. They prefer the status quo, resist change, and make comments like, “we’ve been down this road before,” and “I’ll believe it when I see it.” At best, they’re stubborn “demanders of proof,” willing to believe only when presented with concrete results. At worst, they’re obstructionists—preventing business intelligence initiatives from realizing their full potential.

As BI evolves from traditional reporting and descriptive analytics toward data science and AI, many practitioners fear that new capabilities will make their skill sets obsolete.Fighting new initiatives is, perhaps, a natural preservation instinct. The prevalence of naysayers may also be symptomatic of cultural biases in the institution. Deloitte refers to it as the “inertia of good intentions”—personal behavior created by institutional routines, obligations, and pressures that actually hold many back (unsuspectingly) from delivering the kind of value their organizations need. Left unattended, the culture of most organizations can marginalize BI initiatives to the point of limited and unacceptable return.

You can avoid the negative impact of skeptics and naysayers as well as a culture of mistrust by establishing organizational awareness and building excitement around BI, analytics, and data science initiatives. To amplify means to evangelize.

For instance, large enterprises often create a Data Science Competency Center or AI Center of Excellence, which helps lead the effort to modernize analytics. These evangelists define the data science and AI practices for the firm and are responsible for elevating the general analytical skill level of the entire organization. Fortune 500 Data Science Centers of Excellence are hosting in-depth trainings in data and AI to help bridge the skills gap between the advanced data science practitioners of their organizations and the typical rank-and-file analysts.

Step 4: Qualify

Business intelligence is a journey—a process of continuous improvement meant to adapt and evolve so that business leaders can give intelligent responses to an ever-changing and dynamic business environment. After all, what decision-makers need to monitor and evaluate the business today will change tomorrow. The only way for a business to keep pace is for its reporting and analytics capabilities to keep pace as well.

Today, few firms qualify success properly. They don’t proactively monitor and measure BI performance against end-user expectations and real business outcomes, so they can’t effectively evolve.

Ensure that you focus adequate attention on active monitoring, evaluation, and adjustment of your organization’s business intelligence capabilities so they’re always aligned with the business’ needs and always responsive to stakeholder expectations.

As companies are looking toward growing their BI, analytics, and data science departments, management is demanding results. All too often, analytics projects fall short because leaders fail to understand the key elements of a successful analytics strategy while creating one. In order to plan and execute successful business intelligence efforts, leaders in this area must adopt these imperatives. By focusing your organization’s BI initiative around simplifying, unifying, amplifying, and qualifying business intelligence within the whole organization, you’ll be able to make smarter business decisions, deliver successful results, and keep your firm ahead of the competition.

Continue reading 4 imperatives for making business intelligence work.

Robots, Cryptocurrencies, Bayes, and Brains

1. What People See in a Robot (YouTube) -- In a study using 24 robots selected from this three-dimensional appearance space, I then show that the different dimensions separately predict inferences people make about the robot’s affective, social-moral, and physical capacities. (via RoboHub)
2. Crypto is the Mother of All Scams and (Now Busted) Bubbles While Blockchain Is The Most Over-Hyped Technology Ever, No Better than a Spreadsheet/Database (Nouriel Roubini) -- Roubini's testimony to the Hearing of the U.S. Senate Committee on Banking, Housing and Community Affairs on Blockchains. It is clear by now that Bitcoin and other cryptocurrencies represent the mother of all bubbles, which explains why literally every human being I met between Thanksgiving and Christmas of 2017 asked me first if they should buy them. [...] A chart of Bitcoin prices compared to other famous historical bubbles and scams—like Tulip-mania, the Mississippi Bubble, the South Sea Bubble—shows that the price increase of Bitcoin and other crypto junkcoins was 2X or 3X bigger than previous bubbles, and the ensuing collapse and bust as fast and furious and deeper. [...] Actually calling this useless vaporware garbage a “shitcoin” is a grave insult to manure that is a most useful, precious, and productive good as a fertilizer in agriculture. It's all quotable. Read it.
3. Bayes' Theorem in the 21st Century -- I recently completed my term as editor of an applied statistics journal. Maybe a quarter of the papers used Bayes’ theorem. Almost all of these were based on uninformative priors, reflecting the fact that most cutting-edge science does not enjoy Five-Thirty-Eight-level background information. Are we in for another Bayesian bust?
4. Numenta's New Theory -- research paper, talk, NYT story. Will be interesting to see how this fares in peer review.

Continue reading Four short links: 15 October 2018.

Activity Alert, JavaScript Visualizations, OT vs. CRDT, and Senior Engineering

1. Publicly Available Tools Seen in Cyber Incidents Worldwide (US-CERT) -- The tools detailed in this activity alert fall into five categories: remote access trojans (RATs), webshells, credential stealers, lateral movement frameworks, and command and control (C2) obfuscators. This activity alert provides an overview of the threat posed by each tool, along with insight into where and when it has been deployed by threat actors. Measures to aid detection and limit the effectiveness of each tool are also described. The activity alert concludes with general advice for improving network defense practices.
2. Muze -- Tableau-like visualizations in JavaScript. Open source (MIT).
3. Real Differences between OT and CRDT for Co-Editors -- key CRDT design issues include designing CRDT-special data structures and schemes for representing and manipulating object sequences, searching and executing identifier-based operations in the object sequence, and conversions between internal identifier-based operations and external position-based operations, which collectively deal with both application-specific and concurrency issues in co-editing. This approach has induced a myriad of CRDT-specific challenges and puzzles, such as the correctness of key CRDT data structures and functional components, tombstone overhead, variable and lengthy identifiers, inconsistent-position-integer-ordering and infinite loop flaws, position-order-violation puzzles, and concurrent-insert-interleaving puzzles.
4. What's a Senior Engineer's Job? (Julia Evans) -- I want to talk here about the work that a senior engineer does.

Continue reading Four short links: 12 October 2018.

This collection of serverless resources will get you up to speed on the basics and best practices.

Whether you’re just getting started with serverless or you have previous experience, you’ll find something useful on this list of serverless resources.

The items on this list were curated by O’Reilly’s editorial experts.

Getting started with serverless

Use this introductory material to get up to speed on the basics of serverless.

An Introduction to Serverless — In this short overview, Mike Roberts introduces the concepts behind serverless architectures.

What is Serverless? — Mike Roberts and John Chapin take you through the serverless landscape—particularly the design considerations, tooling, and approaches to operational management you need to make it work.

Learning Path: Getting Started with Serverless — Sam Newman demonstrates the benefits of serverless and provides overviews of function as a service (FaaS) and back end as a service (BaaS).

Implementing serverless

These resources outline best practices for incorporating serverless into your organization.

Learning Path: Migrating Microservices to Serverless — Sam Newman guides you through the many serverless frameworks that are currently available so you can choose the one that’s best for your organization.

Serverless Ops — Michael Hausenblas explores several use cases where serverless is a great fit—primarily short-running, stateless jobs in event-driven architectures found in mobile or IoT applications.

Going Serverless: Security Outside the Box — Jack Naglieri and Austin Byers explore tools and techniques for successfully building, deploying, and debugging serverless security applications.

Lessons Learned from Operating a Serverless-like Platform at Scale — Sangeeta Narayanan shares insights from operating Netflix’s customizable API, which allows the creation of optimized experiences on 1,000+ devices through a serverless-like platform and experience.

Building and Running Serverless Data Pipelines on AWS — Mike Roberts walks through a real-life example of a platform that was rearchitected to provide increased data capacity, reduced cost, and an improved development cycle time.

Microservice Orchestration for Serverless Computing — Cathy Zhang explains how service graphs address the challenge of creating and managing microservice applications.

Continue reading Learn about serverless with these books, videos, and tutorials.

Kristian Hammond maps out simple rules, useful metrics, and where AI should live in the org chart.

Continue reading Bringing AI into the enterprise: A functional approach to the technologies of intelligence.

Supasorn Suwajanakorn discusses the possibilities and the dark side of building artificial people.

Continue reading Building artificial people: Endless possibilities and the dark side.

Decentralized Applications, Global Startups, Better Shuffling, and Prolog Text

1. Decentralized Applications (MIT) -- interesting course to be taught by Robert T Morris. The goal of 6.S974 is to understand recent efforts in decentralized applications, to learn what the main design trade-offs are, and to identify areas for new research. My spidey-sense is tingling. This has all the hallmarks of one of those courses whose graduates build the next wave of companies and research areas.
2. America Is Losing Its Startup Edge -- ignore the use of percentages and Decline of Roman^W American Empire alarmism, it's the rise of the rest of the world that's fascinating here. While it is true that venture-capital investment in the U.S. continues to rise, having reached more than $90 billion in 2017, such investment is growing even faster in other parts of the world, expanding by nearly 375%—more than twice the 160% increase here. China saw the largest jump, its share expanding from 4% of global venture investment in 2005 to a nearly a quarter of it by 2017.
3. Playlist Shuffle -- This paper proposes a novel approach at shuffling a looping sequence that minimizes caveats of naive solutions, keeps computation low, and offers a high degree of variance. [...] The problem is how to repeatedly shuffle a cyclic list and avoid too close and too far duplicates.
4. Art of Prolog, 2E -- this 1994 classic is now an open access title, free PDF download. Prolog is rational AI magic, while deep learning is intuitive AI magic.

Continue reading Four short links: 11 October 2018.

Marc Warner and Louis Barson discuss the internal and external uses of AI in the UK government.

Continue reading Fireside chat with Marc Warner and Louis Barson.

Jason Knight offers an overview of the state of the field for scaling training and inference across distributed systems.

Continue reading Deep learning at scale: A field manual.

Cassie Kozyrkov shares machine learning lessons learned at Google and explains what they mean for applied data science.

Continue reading The missing piece.

Drawing on the McKinsey Global Institute’s research, Michael Chui explores commonly asked questions about AI and its impact on work.

Continue reading Notes from the frontier: Making AI work.

The O’Reilly Data Show Podcast: Jacob Ward on the interplay between psychology, decision-making, and AI systems.

In this episode of the Data Show, I spoke with Jacob Ward, a Berggruen Fellow at Stanford University. Ward has an extensive background in journalism, mainly covering topics in science and technology, at National Geographic, Al Jazeera, Discovery Channel, BBC, Popular Science, and many other outlets. Most recently, he’s become interested in the interplay between research in psychology, decision-making, and AI systems. He’s in the process of writing a book on these topics, and was gracious enough to give an informal preview by way of this podcast conversation.

Continue reading How social science research can inform the design of AI systems.

Ben Lorica and Roger Chen highlight recent trends in data, compute, and machine learning.

Continue reading The state of automation technologies.

Ruchir Puri explains why trust and transparency are essential to AI adoption.

Continue reading Trust and transparency of AI for the enterprise.

Amy Heineike explains how Primer created a self-updating knowledge base that can track factual claims in unstructured text.

Continue reading Why we built a self-writing Wikipedia.