7.0 KiB
- Mdnotes File Name: sculleyHiddenTechnicalDebt
Green Annotations (18/12/2020, 18:46:01)
"Undeclared Consumers. Oftentimes, a prediction from a machine learning model ma is made widely accessible, either at runtime or by writing to files or logs that may later be consumed by other systems. Without access controls, some of these consumers may be undeclared, silently using the output of a given model as an input to another system. In more classical software engineering, these issues are referred to as visibility debt [13]." (Sculley et al :10)
"It may be surprising to the academic community to know that only a tiny fraction of the code in many ML systems is actually devoted to learning or prediction - see Figure 1. In the language of Lin and Ryaboy, much of the remainder may be described as "plumbing" [11]." (Sculley et al :12)
"Using generic packages often results in a glue code system design pattern, in which a massive amount of supporting code is written to get data into and out of general-purpose packages." (Sculley et al :13)
"Because a mature system might end up being (at most) 5% machine learning code and (at least) 95% glue code, it may be less costly to create a clean native solution rather than re-use a generic package." (Sculley et al :13)
"An important strategy for combating glue-code is to wrap black-box packages into common API's. This allows supporting infrastructure to be more reusable and reduces the cost of changing packages." (Sculley et al :13)
"he resulting system for preparing data in an ML-friendly format may become a jungle of scrapes, joins, and sampling steps, often with intermediate files output." (Sculley et al :13)
"Managing these pipelines, detecting errors and recovering from failures are all difficult and costly [1]." (Sculley et al :13)
"Abstraction Debt." (Sculley et al :13)
"The above issues highlight the fact that there is a distinct lack of strong abstractions to support ML systems." (Sculley et al :13)
"What is the right interface to describe a stream of data, or a model, or a prediction?" (Sculley et al :13)
THAT'S THE KEY!!!! (note on p.13)
"distributed learning in particular, there remains a lack of widely accepted abstractions." (Sculley et al :13)
DISTRIBUTED LEARNING (note on p.13)
"The lack of standard abstractions makes it all too easy to blur the lines between components." (Sculley et al :14)
"using multiple languages often increases the cost of effective testing and can increase the difficulty of transferring ownership to other individuals." (Sculley et al :14)
THAT SUPPORTS THE NEED FOR DOMAIN-SPECIFIC LANGUAGES (note on p.14)
"Another potentially surprising area where debt can accumulate is in the configuration of machine learning systems. Any large system has a wide range of configurable options, including which features are used, how data is selected, a wide variety of algorithm-specific learning settings, potential preor post-processing, verification methods, etc." (Sculley et al :14)
"n a mature system which is being actively developed, the number of lines of configuration can far exceed the number of lines of the traditional code. Each configuration line has a potential for mistakes." (Sculley et al :14)
"It should be easy to specify a configuration as a small change from a previous configuration. • It should be hard to make manual errors, omissions, or oversights. • It should be easy to see, visually, the difference in configuration between two models. • It should be easy to automatically assert and verify basic facts about the configuration: number of features used, transitive closure of data dependencies, etc. • It should be possible to detect unused or redundant settings. • Configurations should undergo a full code review and be checked into a repository." (Sculley et al :14)
Support for domain-specific language!!!! (note on p.14)
"Thus if a model updates on new data, the old manually set threshold may be invalid. Manually updating many thresholds across many models is time-consuming and brittle. One mitigation strategy for this kind of problem appears in [14], in which thresholds are learned via simple evaluation on heldout validation data." (Sculley et al :15)
"Unit testing of individual components and end-to-end tests of running systems are valuable, but in the face of a changing world such tests are not sufficient to provide evidence that a system is working as intended." (Sculley et al :15)
"data replaces code in ML systems" (Sculley et al :15)
"code should be tested, then it seems clear that some amount of testing of input data is critical to a well-functioning system" (Sculley et al :15)
Adversial Machine Learning??? (note on p.15)
"non-determinism inherent in parallel learning" (Sculley et al :16)
"Most of the use cases described in this paper have talked about the cost of maintaining a single model, but mature systems may have dozens or hundreds of models running simultaneously [14, 6]." (Sculley et al :16)
THIS IS IMPORTANT ASPECTS TO BE MENTIONED AS MOTIVATION/CHALLENGE (note on p.16)
"How easily can an entirely new algorithmic approach be tested at full scale?" (Sculley et al :16)
"maintainable ML" (Sculley et al :16)
"better abstractions" (Sculley et al :16)
"testing methodologies" (Sculley et al :16)
"design patterns" (Sculley et al :16)