4.2 KiB
4.2 KiB
full-title:: IngestAI/deepmark: Deepmark AI enables a unique testing environment for language models (LLM) assessment on task-specific metrics and on your own data so your GenAI-powered solution has predictable and reliable performance. site:: GitHub labels:: PROJECTS/MOSAICO date-saved:: 22-11-2023 date-archived:: 10-01-2024 is-archived:: 10 source:: Omnivore state:: archived tags:: PROJECTS/MOSAICO Benchmarks
- ### Highlights
- > Deepmark AI empowers Generative AI builders to make informed decisions when choosing among Large Language Models (LLM), enabling seamless assessment of various LLM on your own data, so your AI applications have predictable and reliable performance. [⤴️](https://omnivore.app/me/ingest-ai-deepmark-deepmark-ai-enables-a-unique-testing-environm-18bf7591eb9#f595a922-f1b1-42d5-8354-fe6dfd39bb32)
omnivore-note:: This is very related to our repository #ideas ! #omnivore-note-color
- > Introduction [⤴️](https://omnivore.app/me/ingest-ai-deepmark-deepmark-ai-enables-a-unique-testing-environm-18bf7591eb9#8916d6d4-ad24-4a53-8487-8ee7f88e8f78)
- > legal, ethical and trust issues with Gen AI. These challenges beg the need for a good assessment of the products including metrics that need to aim to improve or rank these various models that drive the overall technology. This is also a roadblock for adaptation of GenAI in several companies today. [⤴️](https://omnivore.app/me/ingest-ai-deepmark-deepmark-ai-enables-a-unique-testing-environm-18bf7591eb9#d14db8d9-0439-4847-a946-5d38e8663023)
omnivore-note:: #challenges #llms #omnivore-note-color
- > Generative AI cannot operate on a set-it-and-forget-it basis — the tools need constant oversight. [⤴️](https://omnivore.app/me/ingest-ai-deepmark-deepmark-ai-enables-a-unique-testing-environm-18bf7591eb9#b314b268-53b5-4ce7-9dbd-6bf595f527af)
- > The only solution close to it is the LangChain LangSmith, which is still in closed beta and is not mature enough to provide comprehensive extrinsic metrics that are essential for adoption. [⤴️](https://omnivore.app/me/ingest-ai-deepmark-deepmark-ai-enables-a-unique-testing-environm-18bf7591eb9#fe6d96bf-75e8-4b89-863e-b2183cf8ca77)
- > organizations need to be able to assess LLM models on their own data to deliver verifiable results that balance accuracy, precision, recall (the model’s ability to correctly identify positive cases within a given dataset), and reliability, as models can produce different answers to the same prompts, impeding the user’s ability to assess the accuracy of outputs. [⤴️](https://omnivore.app/me/ingest-ai-deepmark-deepmark-ai-enables-a-unique-testing-environm-18bf7591eb9#13111341-c7b7-428f-9950-92029c99058b)
omnivore-note:: That's crucial!!!! #STAR #omnivore-note-color
- > benchmarking tool that enables assessment of large language models (LLM) on various extrinsic (task-specific) metrics on your own data. [⤴️](https://omnivore.app/me/ingest-ai-deepmark-deepmark-ai-enables-a-unique-testing-environm-18bf7591eb9#fd01da74-bfff-44ad-9724-93d2ba68b0cf)
- > These assessment metrics are not exhaustive, and specific applications may have additional or alternative metrics depending on the context and requirements, but some of the task-specific metrics like latency, accuracy, or cost can be considered as the most commonly used. [⤴️](https://omnivore.app/me/ingest-ai-deepmark-deepmark-ai-enables-a-unique-testing-environm-18bf7591eb9#fe639712-f0b9-42f1-8bf6-ee854ca3e129)
- > Deepmark AI offers capabilities for comprehensive assessment of various important GenAI performance metrics, such as:
>
* Question answering accuracy
* Text classification accuracy
* PII recognition accuracy
* Named entity recognition (NER) accuracy
* Summarization quality (Relevance)
* Sentiment analysis accuracy
* Cost analysis
* Failure rate
* Accuracy
* Latency [⤴️](https://omnivore.app/me/ingest-ai-deepmark-deepmark-ai-enables-a-unique-testing-environm-18bf7591eb9#972fa4cc-d687-4d04-b1b2-c4a62008a209)