logseq/pages/hls__managingDataIntensiveEcosystems-chapter_1669986389575_0.md

file:: managingDataIntensiveEcosystems-chapter_1669986389575_0.pdf file-path:: ../assets/managingDataIntensiveEcosystems-chapter_1669986389575_0.pdf

• e elaborate on the recent research effort devoted to the mining, analyzing, and evolving data-intensive software ecosystems ls-type:: annotation hl-page:: 1 hl-color:: green id:: 6389f972-b49a-44e1-acef-4887efe6f7d9
• automated support ls-type:: annotation hl-page:: 1 hl-color:: green id:: 6389f979-4478-452d-8bec-d59292e278c1
• Data-intensive software ecosystems comprise one or several databases and a collection of applications connected with the former ls-type:: annotation hl-page:: 2 hl-color:: green id:: 638b0ccc-385c-4366-bdf9-f63f17c29661 hl-stamp:: 1670057166296
• it is the database’s responsibility to handle the query with its best performance ls-type:: annotation hl-page:: 2 hl-color:: green id:: 638b0cf5-2a37-412e-8281-6a320d9ce449
• the increasing use of NoSQL databases poses new challenges for developers and researcher ls-type:: annotation hl-page:: 2 hl-color:: green id:: 638b0d0c-a7e6-48de-b01d-edd1f457fb8c
• hybrid ls-type:: annotation hl-page:: 2 hl-color:: green id:: 638b0d1b-c0d0-4b7a-af5b-d9dd34238158
• hybrid polystores ls-type:: annotation hl-page:: 2 hl-color:: green id:: 638b0d1e-6115-463a-bdd5-ce2458135ffd
• We present recent research initiatives aiming to address those challenges ls-type:: annotation hl-page:: 2 hl-color:: green id:: 638b0d2b-5c16-4390-a29f-62eb0c47ada5
• discuss mining techniques to determine how data is stored and managed in a data-intensive ecosystem ls-type:: annotation hl-page:: 2 hl-color:: green id:: 638b0d35-d1fd-4e4d-9484-17c19ce17509
• static analysis and visualization techniques that exploit the mined information on the storage and manipulation of the ecosystem data ls-type:: annotation hl-page:: 2 hl-color:: green id:: 638b0d4e-10df-4dfb-a1d9-c83cd776fcc4
• findings of empirical studies related to data-intensive software ecosystems ls-type:: annotation hl-page:: 2 hl-color:: green id:: 638b0d5d-e6f6-4fd3-b0c2-dbfed75a8308
• 2.1 Introduction ls-type:: annotation hl-page:: 2 hl-color:: yellow id:: 638b1330-a66f-4fbd-8308-3e24b3505f97 hl-stamp:: 1670058804184
• A subsystem may rely on one or more databases and their data will likely travel through the entire ecosystem. For maintaining and evolving this interconnected system network, it is fundamental to understand how data is handled all over it. ls-type:: annotation hl-page:: 2 hl-color:: green id:: 638b15bc-87f0-42f4-ae17-7d364cb53bf6 hl-stamp:: 1670059455134
• approaches to mining how data is stored and managed in a data-intensive ecosystem. Such knowledge can serve various purposes, e.g., reverse engineering, re-documentation, visualization, or quality assurance approaches ls-type:: annotation hl-page:: 3 hl-color:: green id:: 638b15de-480f-489b-85a7-71155777af54
• We present two techniques to study the interaction points in the applications where they communicate with databases. ls-type:: annotation hl-page:: 3 hl-color:: yellow id:: 638b15ed-79e5-4ff4-994c-9062c9857199 hl-stamp:: 1670059503856
• object-oriented languages usually follow the DAO (Data Access Object) design pattern to isolate the application/business layer from the persistence laye ls-type:: annotation hl-page:: 3 hl-color:: green id:: 638b1684-bca4-4a47-8cbf-5097fa38b4b0 hl-stamp:: 1670059655030
• DAO class implements all the functionality required for fetching, updating, and removing its domain objects ls-type:: annotation hl-page:: 3 hl-color:: green id:: 638b16b3-d511-4e2b-be5f-c41320b8a352
• many systems use combinations of multiple libraries ls-type:: annotation hl-page:: 3 hl-color:: green id:: 638b16e6-d8ed-4308-952f-b02393b68341
• hese libraries can partly or completely hide the actual SQL queries executed by the programs, generating queries at runtime before sending them to the database server ls-type:: annotation hl-page:: 3 hl-color:: green id:: 638b1736-ad72-417d-b1db-9ad4f809100b
• Statement.execute(...) ls-type:: annotation hl-page:: 3 hl-color:: green id:: 638b2d33-bf7e-402a-ba39-0bf7f4be6618
• identify the source code locations where database queries are executed and extract the set of actual SQL queries for each location ls-type:: annotation hl-page:: 4 hl-color:: green id:: 638b2d88-428c-43ec-a7ce-0132bf1b1e7b
• the extraction process balances precision and the computation overhead of in-depth static analyses ls-type:: annotation hl-page:: 6 hl-color:: green id:: 638b2ddc-89ef-4e37-a330-b7c7a140e42d
• hey analyzed the evolution of the same three systems as they have been developed for more than seven years. ls-type:: annotation hl-page:: 6 hl-color:: green id:: 638b2dfa-e45b-4298-9463-455ab228e899
• schema-less storage allows faster data structure change ls-type:: annotation hl-page:: 6 hl-color:: green id:: 638b9415-f580-413a-9136-eb79b085e269 hl-stamp:: 1670091801940
• multiple ls-type:: annotation hl-page:: 6 hl-color:: green id:: 638b9442-98db-427f-8081-61be8efb6379
• co-existing implicit schemas ls-type:: annotation hl-page:: 6 hl-color:: green id:: 638b9446-8220-44c7-845e-39cb3df65e80
• support schema evolution in the schema-less NoSQL environmen ls-type:: annotation hl-page:: 6 hl-color:: green id:: 638b947b-7c4e-41ae-9825-4fe8a7f3755c
• schema extractio ls-type:: annotation hl-page:: 6 hl-color:: green id:: 638b9487-a71a-4c69-ae1b-bdfb29990f6e
• schema genera- ls-type:: annotation hl-page:: 6 hl-color:: green id:: 638b948a-d603-407e-96a1-2151ee52fc2d
• optimization ls-type:: annotation hl-page:: 6 hl-color:: green id:: 638b948c-af44-49aa-9c3b-f721a86e17a8
• allenges of NoSQL systems. A popular one is to support schema evolution in the schema-less NoSQL environment [12]. For example, researchers study automatic schema extraction [13], schema generation [14], optimization [15], and s ls-type:: annotation hl-page:: 6 hl-color:: green id:: 638b9490-bc1b-4652-ab79-50524392a68d
• ema suggestions [16]. Behind the scenes, such approaches mainly rely on a static analysis of the sou ls-type:: annotation hl-page:: 6 hl-color:: green id:: 638b9495-bbf1-4b0d-930f-39c9d06b0fc2
• andle the excessively dynamic features of the languag ls-type:: annotation hl-page:: 7 hl-color:: green id:: 638b94d5-5d8e-4865-b49a-6067089ab001
• h type inferenc ls-type:: annotation hl-page:: 7 hl-color:: green id:: 638b94da-ded3-4edd-af78-29438e30cc67
• data flo ls-type:: annotation hl-page:: 7 hl-color:: green id:: 638b94dc-6b1f-4193-8727-33c1ba2eb19c
• graphs ls-type:: annotation hl-page:: 7 hl-color:: green id:: 638b94df-25a6-453c-a6e3-ae10a235b365
• call ls-type:: annotation hl-page:: 7 hl-color:: green id:: 638b94e2-d06e-41c0-987f-329c7ab673eb
• r e q u i r e ( " mongoose " ) ; ls-type:: annotation hl-page:: 7 hl-color:: green id:: 638bda2b-cba2-444d-98b1-e16e31547976
• Schema is mapped to a MongoDB collection and defines the structure of the documents within that collection ls-type:: annotation hl-page:: 7 hl-color:: green id:: 638bda69-4d7e-4f8b-b4c1-807b48288907
• CodeQL,3 a code analysis engine developed by GitHu ls-type:: annotation hl-page:: 8 hl-color:: green id:: 638dea05-b734-42cf-ba67-af7596d43005
• We presented two static analysis approaches to study how applications communicate with their databases. ls-type:: annotation hl-page:: 9 hl-color:: green id:: 638deb12-a99e-444b-bd76-6aeaad13b8a3
• Brink et al. present a quality assessment approach for SQL statements embedded in PL/SQL, COBOL, and Visual Basic code [ ls-type:: annotation hl-page:: 10 hl-color:: green id:: 638deb40-0b44-435b-b4a8-0bbf70907f8c
• we show static analysis approaches in Section 3.2 ls-type:: annotation hl-page:: 11 hl-color:: green id:: 638debc6-2a34-4970-9460-d8a86f1689be
• dependencies between the database and different components of an ecosystem ls-type:: annotation hl-page:: 11 hl-color:: green id:: 638debd4-f758-4129-82c6-4905b8dd4727
• because the application code and the database depend on each other, they evolve in parallel [8], resulting in increased complexity of the database communication code. ls-type:: annotation hl-page:: 11 hl-color:: green id:: 638dec2a-f439-463a-b203-d9877383dd22
• SQLInspect ls-type:: annotation hl-page:: 11 hl-color:: green id:: 638dec44-23ff-4e7a-a9fd-5ae1049bdf3e
• A lightweight analyzer can pinpoint a mistake already in the IDE before the developer commits it ls-type:: annotation hl-page:: 11 hl-color:: green id:: 638dec55-f0e7-461c-80d9-3bcd782aa073
• estimate the effort required in the past for adapting the applications to database schema changes ls-type:: annotation hl-page:: 13 hl-color:: green id:: 638ded2c-cf12-4c80-8695-5f46a662aa3f
• hey rely on the previous analysis approach w ls-type:: annotation hl-page:: 13 hl-color:: green id:: 638ded56-fbf9-4c36-9b42-a7ff0d7dd641
• information visualization field, ls-type:: annotation hl-page:: 13 hl-color:: green id:: 638dee0b-a1bc-4832-8919-305b72fc7be2
• visualization is the preferred way of getting acquainted with large data ls-type:: annotation hl-page:: 13 hl-color:: green id:: 638dee14-e0fe-4830-8c50-bde4f328bb66
• As we could see in the previous example, the city metaphor for visualizing software systems in 3D has been widely explored and has led to various implementations and approaches ls-type:: annotation hl-page:: 16 hl-color:: green id:: 638df06f-0f30-4dcf-a5d9-7d7dc8a91631
• he resulting view seamlessly integrates data sources into a software city and enables a comprehensive understanding of a system’s source code and data. ls-type:: annotation hl-page:: 16 hl-color:: green id:: 638e28f3-eba8-4572-887b-30397eab9037
• static analysis approach to identify SQL code smells in the database communication layer of data-intensive application ls-type:: annotation hl-page:: 17 hl-color:: green id:: 638e2990-90d3-4a2a-89ed-c3a5459f611d
• a visualization approach of database behavio ls-type:: annotation hl-page:: 18 hl-color:: green id:: 638e2a05-1563-4813-b432-011e4ce0af27
• 4 Empirical Studies ls-type:: annotation hl-page:: 19 hl-color:: green id:: 638e2a2c-065e-468f-87ad-a0625628cc60
• Libraries.io ls-type:: annotation hl-page:: 19 hl-color:: green id:: 638e2bce-6955-4294-817a-ffa094cf5ac2
• 40,609 projec ls-type:: annotation hl-page:: 19 hl-color:: green id:: 638e2be7-2942-4fdf-9ecc-57eabdafa4f9
• However, the popularity of SQL technologies has recently decreased concerning NoSQL datastores ls-type:: annotation hl-page:: 19 hl-color:: green id:: 638e2bf8-24be-4141-b33c-cb22f44e2f32
• more than 56% of systems relying on a key-value database also use another technology, typically relational or document-oriented. ls-type:: annotation hl-page:: 19 hl-color:: green id:: 638e2c1a-13a4-4716-b1c8-695a7fb5d1b9
• revalence, Impact and Evolution of SQL Bad Smells ls-type:: annotation hl-page:: 21 hl-color:: green id:: 638e2c48-923f-447a-846f-f588420318c0
• they further examined the 72 projects for which they could execute the tests and collect coverage reports. ls-type:: annotation hl-page:: 22 hl-color:: green id:: 638e2d52-582f-4984-9dcb-cf3b8a10ee4c
• Non DB methods test coverage rate ls-type:: annotation hl-page:: 22 hl-color:: green id:: 638e2d62-a0be-4741-b382-c2951241651e
• DB access methods test coverage rate ls-type:: annotation hl-page:: 22 hl-color:: green id:: 638e2d67-1393-456a-b7cb-01e57f8fdc71
• The results indicate that the database manipulation code was poorly tested: 46% of the projects did not test half of their DB methods, and 33% did not test DB communication. ls-type:: annotation hl-page:: 22 hl-color:: green id:: 638e2dce-2fe3-4753-a5a3-560b5b2d0a82
• This chapter summarized the recent research efforts devoted to mining, analyzing and evolving data-intensive software ecosystems. ls-type:: annotation hl-page:: 26 hl-color:: green id:: 638e2e04-e975-4500-9586-d3d57e619fdf
• (1) both the databases and the programs are essential ecosystem artifacts ls-type:: annotation hl-page:: 26 hl-color:: green id:: 638e2e18-c878-4bc6-82f0-be9f1b040b0b
• (2) mining, analyzing and visualizing what the programs are doing on the data may considerably help in understanding the system in general, and the databases in particular, ls-type:: annotation hl-page:: 26 hl-color:: green id:: 638e2e21-7b89-4ec0-99c7-98df121af432
• (3) database interactions may suffer from quality problems and technical debt and should be better tested ls-type:: annotation hl-page:: 26 hl-color:: green id:: 638e2e29-dfa7-459c-8a5c-98829e51c895
• (4) software evolution methods should devote more attention to the program-database co-evolution problem ls-type:: annotation hl-page:: 26 hl-color:: green id:: 638e2e2f-89f1-4fa9-935a-0655e8616e76
• 4.6 Reflections ls-type:: annotation hl-page:: 23 hl-color:: red id:: 638e2f92-7895-4037-8c73-41e7df23f499 hl-stamp:: 1670262677877