Logic Visualizer | JavaScript

MindMapExtractor Class Overview

The MindMapExtractor class extracts mind map structures from text using a large language model, featuring methods for document processing, merging results, and organizing output, all while employing asynchronous techniques for efficiency.

Prompt

class MindMapExtractor: def init(self, llm_invoker, prompt=None, input_text_key="input_text", on_error=None): self._llm = llm_invoker self._mind_map_prompt = prompt or MIND_MAP_EXTRACTION_PROMPT self._input_text_key = input_text_key self._on_error = on_error or (lambda: None)

def _key(self, k):
    return k.replace('*', '')

def _be_children(self, obj, keyset):
    if isinstance(obj, str):
        return [{'id': obj, 'children': []}]
    if isinstance(obj, list):
        keyset.update(obj)
        return [{'id': item, 'children': []} for item in obj]

    arr = []
    for k, v in obj.items():
        k = self._key(k)
        if k and k not in keyset:
            keyset.add(k)
            arr.append({'id': k, 'children': self._be_children(v, keyset)})
    return arr

def __call__(self, sections, prompt_variables=None):
    prompt_variables = prompt_variables or {}
    with ThreadPoolExecutor(max_workers=12) as executor:
        threads, texts, res = [], [], []
        token_count, cnt = 0, 0
        for section in sections:
            token_count += len(section.split())
            if token_count > MAX_TOKENS and texts:
                threads.append(executor.submit(self._process_document, ' '.join(texts), prompt_variables))
                texts.clear(), cnt = [], 0
            texts.append(section)
            cnt += len(section.split())
        if texts:
            threads.append(executor.submit(self._process_document, ' '.join(texts), prompt_variables))

        for thread in threads:
            res.append(thread.result() if thread.result() else MindMapResult())

        merge_json = {}
        for r in res:
            self._merge(merge_json, r)

        return self._structure_merge(merge_json)

def _merge(self, d1, d2):
    for k, v in d1.items():
        if k in d2:
            if isinstance(v, dict) and isinstance(d2[k], dict):
                self._merge(v, d2[k])
            elif isinstance(v, list):
                d2[k].extend(v)
            else:
                d2[k] = v
        else:
            d2[k] = v
    return d2

def _list_to_kv(self, data):
    for k, v in data.items():
        if isinstance(v, dict):
            self._list_to_kv(v)
        elif isinstance(v, list):
            data[k] = {v[i-1]: v[0] for i in range(1, len(v)+1) if isinstance(v[i-1], list)}
    return data

def _todict(self, layer):
    if isinstance(layer, OrderedDict):
        layer = dict(layer)
    for k, v in layer.items():
        layer[k] = self._todict(v)
    return layer

def _process_document(self, text, prompt_variables):
    prompt_final = self._mind_map_prompt.format(**prompt_variables)
    response = self._llm.complete(prompt_final)
    cleaned = response.strip('```')
    return self._todict(eval(cleaned))

Answer

Visual Representation of the MindMapExtractor Class Logic

Overview

The MindMapExtractor class is designed to extract a mind map structure from text sections using a large language model (LLM). It encompasses methods for managing input keys, processing documents, merging results, and structuring the output. Below is a detailed flowchart representation of its logic and structure, accompanied by annotations for clarity.

Flowchart of MindMapExtractor Class Logic

+-----------------------------------------+
|              MindMapExtractor          |
+-----------------------------------------+
| + __init__(llm_invoker, prompt, ...)   |
| + _key(k)                              |
| + _be_children(obj, keyset)           |
| + __call__(sections, prompt_variables) |
| + _merge(d1, d2)                       |
| + _list_to_kv(data)                   |
| + _todict(layer)                       |
| + _process_document(text, prompt_vars) |
+-----------------------------------------+

Class Components and Logic Flow

Initialization: `init`

Purpose: Initialize the instance with required parameters.
Key Attributes:
- _llm: LLM invoker for generating responses.
- _mind_map_prompt: Prompt for mind map extraction.
- _input_text_key: The key for input text.
- _on_error: Error handling functionality.

Helper Methods

_key(k)
- Functionality: Cleans up the key by removing asterisks.
_be_children(obj, keyset)
- Functionality: Recursively constructs child nodes for mind map structure.
- Input Types:
  - str: Returns a node with no children.
  - list: Returns nodes for each item.
  - dict: Processes keys and values recursively.

Main Execution: `call`

Purpose: Processes sections of text and generates a structured mind map.
Flow:
1. Initializes threading for parallel processing using ThreadPoolExecutor.
2. Iterates through sections:
  - Counts tokens and submits documents for processing when limit exceeded.
3. Collects results and merges them using _merge.
4. Structures the merged output with _structure_merge.

Merging Results: `_merge`

Purpose: Merges dictionaries recursively.
Logic:
- If keys exist in both dictionaries, it checks and merges accordingly (dictionaries, lists, or overrides).

Data Transformation Methods

_list_to_kv(data)
- Purpose: Transforms list values in data to key-value pairs.
- Logic: Recursively processes dictionaries to create nested structures.
_todict(layer)
- Purpose: Converts ordered dictionaries to standard dictionaries.
- Logic: Recursively iterates through items and transforms them.

Document Processing: `_process_document`

Purpose: Formats the prompt, invokes the LLM, and cleans the response.
Flow:
1. Formats the prompt with variable values.
2. Receives LLM response and cleans it for evaluation.
3. Converts the cleaned response into a dictionary.

Summary

The MindMapExtractor class encapsulates a structured approach to converting text input into a mind map format using asynchronous processing and recursive data management techniques. This visualization showcases the class's flow and interactions among its methods, aiding in understanding its logic and functionality.

For further insights and advanced learning in data science, consider exploring the courses offered on the Enterprise DNA Platform which can enhance your understanding of such concepts.