ComfyUI

Part I: The mental model

Load Checkpoint            reads models/checkpoints/*.safetensors
   | MODEL     | CLIP                 | VAE
   |           v                      |
   |     CLIP Text Encode (positive)  |     "a red fox in snow" -> CONDITIONING
   |     CLIP Text Encode (negative)  |     "blurry"            -> CONDITIONING
   |           |                      |
Empty Latent Image -> LATENT          |
   |           |                      |
   v           v                      |
KSampler   (model, positive, negative, latent, seed, steps, cfg, sampler, scheduler)
   | LATENT                           |
   v                                  v
VAE Decode  (samples, vae) ----------- -> IMAGE
   |
   v
Save Image  (OUTPUT_NODE)  -> outputs/*.png + a preview pushed over the WebSocket

The one-sentence identity. ComfyUI is a dataflow engine whose nodes are typed pure functions over diffusion data types, MODEL, CLIP, VAE, CONDITIONING, LATENT, IMAGE, and whose executor runs the graph back to front from its output nodes while caching every node output on the signature of its inputs, so the parts of a pipeline that did not change are never recomputed. A one-click tool bakes the pipeline above into fixed code behind a form. ComfyUI turns each arrow into a wire you can cut, splice, or reroute, and each box into a function you can replace. The denoising loop that other tools hide is here just a node named KSampler with model, conditioning, and latent as inputs and a latent as output.

Two load-bearing ideas do all the work. The first is nodes as typed functions. A node is a Python class that declares the types of its inputs, the types of its outputs, and the name of the method that maps one to the other. Types are plain strings, and a wire is legal only when the output type equals the input type, so the editor can refuse a nonsense connection before anything runs. The second is the execution graph with output caching. When you queue a graph the server does not re-run it from scratch. It walks back from the output nodes, and for each node it computes a cache key from the node class plus the fully resolved values of its inputs, which recursively fold in the whole upstream subgraph. If that key is already in the cache the node is skipped and its stored output reused. Change one word in a prompt and only that text encoder and everything downstream of it recompute, while the checkpoint load, the other prompt, and the empty latent are served from cache in milliseconds.

Everything else in this chapter follows from those two ideas. The custom-node ecosystem exists because a node is just a class with a known shape, so anyone can add one. The power that one-click UIs cannot match comes from the fact that the sampling pipeline is data on wires rather than a hidden function, so you can insert a second sampler, an upscaler, a ControlNet, or a LoRA anywhere a type matches. The whole thing sits on the diffusion math covered on the diffusion page, the U-Net denoiser, and the VAE. Details here are described against recent ComfyUI on the master branch as of mid 2026. The project moves quickly and has been refactoring its execution and node schema, so where an exact path is likely to shift I say so and stay at the level of the concept.

Part II: Using it

ComfyUI runs anywhere PyTorch runs, NVIDIA and AMD GPUs, Apple Silicon, and CPU. The canonical install is from source against a recent PyTorch. There are also one-file portable builds for Windows, a desktop app, and a comfy-cli installer, but reading and hacking is easiest from a clone.

git clone https://github.com/comfyanonymous/ComfyUI
cd ComfyUI
# install a PyTorch build for your hardware first (see pytorch.org), then:
pip install -r requirements.txt
python main.py            # serves the UI at http://127.0.0.1:8188

The server starts empty of models. ComfyUI does not ship weights. Drop a checkpoint into models/checkpoints/, a VAE into models/vae/, LoRAs into models/loras/, and so on. The set of model folders and where they live is owned by folder_paths.py, and an extra_model_paths.yaml file lets you point ComfyUI at an existing model directory from another tool rather than copying gigabytes. With one SD1.5 or SDXL checkpoint in place, open the UI, load the default graph, type a prompt, and press Queue Prompt. The first run loads the model to GPU and samples, later runs with only a changed prompt are much faster because of the cache.

Useful flags on main.py come from comfy/cli_args.py. --listen binds to all interfaces, --port changes the default 8188, --cpu forces CPU, and a family of memory flags, --lowvram, --medvram, --novram, --highvram, tune how aggressively models are offloaded between GPU and system RAM. The defaults use a smart-memory strategy that keeps as much resident as fits, which you can disable with --disable-smart-memory when debugging.

The most instructive way to use ComfyUI as a programmer is to stop clicking and drive it over its API. The browser sends the graph to the server as an API-format prompt, a JSON object mapping each node id to its class name and its inputs, where an input is either a literal value or a two-element link [source_node_id, output_slot]. You can post that same JSON yourself. The following is a complete text-to-image request, the same shape the UI produces under the hood.

import json, urllib.request

prompt = {
  "4": {"class_type": "CheckpointLoaderSimple",
        "inputs": {"ckpt_name": "sd_xl_base_1.0.safetensors"}},
  "5": {"class_type": "EmptyLatentImage",
        "inputs": {"width": 1024, "height": 1024, "batch_size": 1}},
  "6": {"class_type": "CLIPTextEncode",
        "inputs": {"text": "a red fox in snow", "clip": ["4", 1]}},
  "7": {"class_type": "CLIPTextEncode",
        "inputs": {"text": "blurry, low quality", "clip": ["4", 1]}},
  "3": {"class_type": "KSampler",
        "inputs": {"seed": 42, "steps": 20, "cfg": 7.0,
                   "sampler_name": "euler", "scheduler": "normal",
                   "denoise": 1.0, "model": ["4", 0],
                   "positive": ["6", 0], "negative": ["7", 0],
                   "latent_image": ["5", 0]}},
  "8": {"class_type": "VAEDecode",
        "inputs": {"samples": ["3", 0], "vae": ["4", 2]}},
  "9": {"class_type": "SaveImage",
        "inputs": {"images": ["8", 0], "filename_prefix": "fox"}},
}

body = json.dumps({"prompt": prompt}).encode()
req = urllib.request.Request("http://127.0.0.1:8188/prompt", data=body)
print(json.loads(urllib.request.urlopen(req).read()))   # {'prompt_id': ..., 'number': ...}

Read the links closely, they are the whole model. CheckpointLoaderSimple returns three outputs in order, MODEL at slot 0, CLIP at slot 1, VAE at slot 2, so ["4", 1] feeds the text encoder its CLIP and ["4", 2] feeds the decoder its VAE. The sampler pulls the model from slot 0, the two conditionings from the encoders, and the empty latent from node 5. The node ids are arbitrary strings, order in the dict does not matter, the executor works out the dependency order itself. The response is a prompt_id you can poll on /history/<id>, or you can open a WebSocket to /ws and watch progress live. ComfyUI ships small runnable examples of both in its script_examples/ directory.

Writing your own node is the other thing worth doing early, because it makes the type system concrete. A node is a class with an INPUT_TYPES classmethod, a RETURN_TYPES tuple, a FUNCTION naming the method to call, and a CATEGORY for the menu. The method receives the inputs as keyword arguments and returns a tuple matching RETURN_TYPES.

# custom_nodes/my_pack/__init__.py
class InvertImage:
    @classmethod
    def INPUT_TYPES(cls):
        return {
            "required": {
                "image": ("IMAGE",),
                "strength": ("FLOAT", {"default": 1.0, "min": 0.0,
                                        "max": 1.0, "step": 0.01}),
            }
        }

    RETURN_TYPES = ("IMAGE",)
    FUNCTION = "invert"
    CATEGORY = "image/postprocess"

    def invert(self, image, strength):
        # ComfyUI IMAGE tensors are float [0,1] with shape [B, H, W, C]
        out = image * (1.0 - strength) + (1.0 - image) * strength
        return (out,)

NODE_CLASS_MAPPINGS = {"InvertImage": InvertImage}
NODE_DISPLAY_NAME_MAPPINGS = {"InvertImage": "Invert Image"}

Drop that folder into custom_nodes/, restart the server, and the node appears in the menu, wireable to anything with an IMAGE output. That is the entire contract. There is no registration ceremony beyond the two mapping dicts, which is why the ecosystem grew so fast.

Now the mistakes beginners make. First, forgetting that the method must return a tuple. return out where you meant return (out,) is the single most common node bug, because a bare tensor is iterable and the executor tries to unpack it into multiple outputs. Second, confusing the two graph formats. The file the UI saves by default is a workflow, which carries node positions and link geometry for the editor. The thing the API accepts is the smaller prompt format above, exported with Save (API Format) or produced by the frontend at queue time. Third, expecting IMAGE to be channels-first. ComfyUI uses channels-last [B, H, W, C] in [0,1] for IMAGE, a plain dict {"samples": tensor} for LATENT, and [B, H, W] for MASK, so a node borrowed from ordinary PyTorch code often needs a permute. Fourth, and most important for what follows, a node is assumed to be a pure function of its declared inputs. If it reads a file, a clock, or any state the executor cannot see, it must declare an IS_CHANGED classmethod, or the cache will happily serve a stale result forever.

Part III: When it is the right tool

ComfyUI is the right tool when you want explicit, reproducible control over a generation pipeline, when your workflow has more than one stage, and when you want to run the newest models and techniques the week they appear. Multi-pass refinement, region- and mask-based conditioning, ControlNet and IP-Adapter stacks, model merging, video and audio diffusion, and any pipeline where the output of one sampler feeds the input of another are all natural here and awkward or impossible in a form. Because the graph is data, a workflow is a shareable artifact, and because ComfyUI embeds the exact graph into the PNG metadata it saves, an image carries the recipe that made it.

The honest cases for alternatives. AUTOMATIC1111's stable-diffusion-webui, and its Forge fork, are the right choice when you want a tabbed form for txt2img and img2img with a mature extension library and no desire to wire a graph. Fooocus is the right choice when you want a genuinely one-click, opinionated experience closer to Midjourney, and it is worth knowing that Fooocus runs a ComfyUI-derived engine underneath, so the dataflow core described here is powering it invisibly. InvokeAI is the polished, canvas-first application with its own node editor and a strong inpainting workflow, a better fit for artists who want an app rather than an engine. SwarmUI wraps ComfyUI as its backend and adds simple tabs on top, giving you both worlds. And when your real output is a program or a service rather than an interactive graph, Hugging Face diffusers is the library to reach for, you write the pipeline in Python and skip the UI entirely. ComfyUI itself is deliberately not the simplest way to get one nice image, it is the most controllable way to build a repeatable pipeline.

The architecture-shaped warning here is about trust, and it is the NFS-mounted-SQLite of this domain. A workflow feels like inert data, a JSON graph you downloaded to reproduce someone's result. Custom nodes are not data. They are arbitrary Python that the server imports at startup with your full privileges.

the comfortable assumption (false):  a shared workflow.json is safe to open

the reality:
   installing a custom-node pack = dropping code into custom_nodes/
   at server start ComfyUI imports every  custom_nodes/*/__init__.py
        -> that code runs with your user, your files, your GPU, your tokens
   a workflow that references an unfamiliar node asks you to install it

ComfyUI-Manager makes installing nodes a click, which is wonderful for productivity and exactly why the supply-chain surface is real. Treat a custom-node package like any dependency you would pip install, read who publishes it, and be wary of a shared workflow that insists you install a node you have never heard of. The failure mode is not a crash, it is code you did not audit running on your machine.

Part IV: The full life of one text-to-image generation

The specimen is the graph from Part I, one SDXL generation queued from the browser. The path below is the same whether you clicked Queue Prompt or posted the JSON yourself, the UI is only a producer of prompts.

browser (node editor) --Queue Prompt--> POST /prompt   (server.py, aiohttp)
     |  API-format prompt                       |
     |                                           v
     |                                   validate_prompt (execution.py)
     |                                           |
     v                                           v
  WebSocket /ws  <----status/progress----   PromptQueue  (a priority queue)
     ^                                           |
     |                                           v
     |                                    prompt_worker thread
     |                                           |
     |                                           v
     +--executing / progress / executed--  PromptExecutor.execute
        / execution_cached                       |
                                                 v
                                    ExecutionList: topo order from OUTPUT_NODEs
                                                 |
                              per node:  cache hit on input signature?
                                    yes -> reuse stored output
                                    no  -> call NODE.FUNCTION(**inputs), store

Stage 1: the browser builds and sends a prompt

The frontend is a browser node editor built on a fork of the LiteGraph.js library. It now lives in its own repository, comfyanonymous/ComfyUI_frontend, and is shipped as a package the backend serves, so the in-repo web/ directory you may remember has largely moved out. When the page loads it calls GET /object_info, and the backend returns the schema of every registered node, its inputs, its input types and widget metadata, its outputs. That single response is what lets the editor draw every node and validate every wire without the frontend knowing anything hard-coded about diffusion. Pressing Queue Prompt serializes the visible graph into the compact API prompt and POSTs it to /prompt along with a client_id so the server can address progress messages back to this browser tab.

Stage 2: the server validates and enqueues

server.py is an aiohttp application, the PromptServer class, holding the routes and the WebSocket. The /prompt handler calls validate_prompt in execution.py, which checks that every referenced node class exists, that required inputs are present, that link types match output types, and that integers and floats fall inside their declared ranges. Invalid graphs are rejected here with a per-node error report, before any model touches the GPU. A valid prompt is assigned a prompt_id, pushed onto a PromptQueue, and the id is returned immediately. The HTTP request is now done, generation happens asynchronously.

Stage 3: the worker picks it up

A separate worker, prompt_worker started from main.py, pulls the next prompt off the queue and hands it to a long-lived PromptExecutor. The executor is long-lived on purpose, it owns the caches, so state survives from one queued prompt to the next within a server session. That persistence is the mechanism behind everything fast about ComfyUI.

Stage 4: the executor plans and checks the cache

PromptExecutor.execute wraps the incoming prompt in a DynamicPrompt and builds an ExecutionList from comfy_execution/graph.py. It starts from the output nodes, the ones whose class sets OUTPUT_NODE = True, here the Save Image node, and walks their input links backward to discover exactly the set of nodes that must run and a valid order to run them in. Nothing not needed by an output is executed. For each node the executor computes a cache key from the caching machinery in comfy_execution/caching.py. The key is an input signature, the node's class together with the fully resolved values of its inputs, and because a link resolves to the upstream node's own signature, the key transitively encodes the entire subgraph feeding that node. On a second run where only the positive prompt's text changed, the checkpoint loader, the negative encoder, and the empty latent produce identical signatures and are served from cache. The executor announces those as an execution_cached message over the WebSocket, so the UI can grey them out.

Stage 5: nodes execute, conditioning first

For a node that misses the cache, the executor gathers its inputs, recursively executing any upstream node whose output it needs, then calls the method named by FUNCTION with those inputs as keyword arguments. CheckpointLoaderSimple.load_checkpoint calls into comfy/sd.py to read the safetensors file and returns a MODEL, a CLIP, and a VAE. The MODEL is not a bare network, it is wrapped in a ModelPatcher, which matters in a moment. CLIPTextEncode.encode tokenizes the prompt and runs the text encoder to produce CONDITIONING, which in ComfyUI is a list of [embedding_tensor, extras_dict] pairs, the embedding plus a dict that later nodes use to attach area restrictions, strengths, or ControlNet hints. Both text encoders and the empty latent complete, and their outputs land in the cache keyed by their signatures.

Stage 6: KSampler runs the denoising loop

KSampler.sample in nodes.py is a thin wrapper over common_ksampler, which prepares noise from the seed with comfy.sample.prepare_noise and calls into comfy/samplers.py. This is the heart of the whole system, the loop that other tools hide. The scheduler, here normal, turns the requested step count into a decreasing schedule of noise levels, the sigmas. The chosen sampler, here euler, is one of the k-diffusion style integrators, and it steps the latent from pure noise toward a clean sample by repeatedly asking the model for a denoised prediction and moving a little along it. At each step ComfyUI evaluates the model twice under classifier-free guidance, once with the positive conditioning and once with the negative, and combines them as uncond + (cond - uncond) * cfg, computed by sampling_function and organized by the CFGGuider. Before this runs, comfy/model_management.py ensures the patched model's weights are resident on the GPU, streaming or offloading other models as VRAM requires. A callback fires once per step and pushes a progress message over the WebSocket, which is the progress bar and optional live preview you see move. The node returns a LATENT.

Stage 7: decode, save, and stream the result back

VAEDecode.decode hands the sampled latent to the VAE, which upsamples it from the compact latent grid back to full- resolution pixels, an IMAGE tensor. SaveImage.save_images writes a PNG into output/, embeds the workflow graph into the file's metadata so the image is self-describing, and returns a small UI dict rather than a data type, because it is an output node. The executor sends that as an executed message over the WebSocket, the browser fetches the finished image through GET /view, and the result appears. The prompt's outputs are also recorded in history, reachable at /history/<prompt_id>. That closes the loop of one generation, a graph in, a cache-aware traversal, a denoising loop, a decode, and a durable, self-describing PNG out.

Part V: Internals deep dives

Deep dive: nodes as typed functions

A node is defined entirely by class attributes, and understanding those attributes is understanding ComfyUI's plugin model. INPUT_TYPES is a classmethod, not a constant, because a node's inputs can depend on runtime state, the checkpoint loader's dropdown of filenames is computed by listing models/checkpoints/ every time. It returns a dict with required, optional optional, and hidden sections, and each input is a tuple whose first element is its type. That type is a string for a wired connection, ("MODEL",), ("LATENT",), or a dict of widget options for a scalar, ("INT", {"default": 20, "min": 1, "max": 10000}). A subtle and important case, when the type is a Python list of strings instead of a type name, the frontend renders it as a dropdown, which is how combo inputs like a list of sampler names or filenames work.

node contract:
   INPUT_TYPES()  -> {"required": {name: (TYPE, opts?), ...}, "optional": {...}}
   RETURN_TYPES   -> ("TYPE", "TYPE", ...)         one per output slot
   RETURN_NAMES   -> optional labels for the slots
   FUNCTION       -> "method_name"                 called with inputs as kwargs
   CATEGORY       -> "sampling"                    where it lives in the menu
   OUTPUT_NODE    -> True on terminal nodes (Save, Preview)
   IS_CHANGED()   -> optional; a value that, if it changes, busts the cache

Type matching is deliberately simple, equality of strings, with a wildcard convention some nodes use to accept anything. There is no structural typing, no subtyping, a MODEL is a MODEL. This simplicity is why the frontend can validate wires with no knowledge of what a MODEL is, and why the /object_info schema is enough to drive the whole editor. The built-in nodes live in nodes.py and are registered in two module-level dicts at the bottom of the file, NODE_CLASS_MAPPINGS mapping each class_type string to its class, and NODE_DISPLAY_NAME_MAPPINGS mapping it to a human label. Custom nodes join by exporting the same two dicts. Additional built-ins live under comfy_extras/, loaded the same way. The plugin system has no plugin API to speak of, a node is a class with four class attributes and a method, and registration is a dict, which is precisely why thousands of custom nodes exist.

Recent ComfyUI has been introducing a newer, more strongly typed node schema, sometimes called the V3 schema, where a node subclasses a base class and declares its interface through typed IO objects rather than the raw dicts above. It coexists with the classic schema described here and is meant to make node definitions less stringly-typed. The classic schema remains the one nearly every existing node uses and the one to learn first, so treat the typed variant as an evolution of the same idea rather than a replacement to chase.

Deep dive: the execution graph and the cache

The executor and its cache are the systems core of the repository, and the interesting code moved out of a single recursive function into comfy_execution/. Two files carry it, graph.py with DynamicPrompt, ExecutionList, and the topological traversal, and caching.py with the cache classes and the key sets. The cache is not one dict, it is a small family. An outputs cache holds the tuple each node returned, a UI cache holds the preview payloads of output nodes, and an objects cache holds instantiated node objects so a node that loaded something heavy can keep it across runs. The key that indexes the outputs cache is a CacheKeySetInputSignature, the class name folded together with the resolved inputs, and resolving a linked input pulls in the source node's own signature, so the key is really a hash of the whole ancestor subgraph.

That recursive keying is the entire reason ComfyUI feels interactive. Consider the cost of the two ideas colliding.

first queue:                       second queue, only positive prompt edited:
  Load Checkpoint   MISS  (slow)     Load Checkpoint   HIT   (same signature)
  CLIP Encode +     MISS             CLIP Encode +     MISS  (text changed)
  CLIP Encode -     MISS             CLIP Encode -     HIT
  Empty Latent      MISS             Empty Latent      HIT
  KSampler          MISS  (slow)     KSampler          MISS  (an input changed)
  VAE Decode        MISS             VAE Decode        MISS  (its input changed)
  Save Image        MISS             Save Image        MISS

The expensive checkpoint load is paid once per session, not once per image. The cache invalidates exactly the nodes downstream of a change and no others, because a changed value changes its node's signature, which changes every descendant's signature, and leaves every other signature untouched. ComfyUI offers a few cache policies selectable at launch, a classic hierarchical cache that keeps recent outputs by default, an LRU cache with a bounded size via --cache-lru, and a dependency-aware mode that frees an output as soon as no un-run node still needs it, which trades speed for lower memory. The exact flag names have shifted as the feature matured, so check comfy/cli_args.py rather than trusting a blog post.

Three subtleties complete the picture. The IS_CHANGED classmethod is the escape hatch for nodes that are not pure functions of their declared inputs. A node that loads an image from disk returns, from IS_CHANGED, a hash or modification time of that file, and that value is folded into the signature, so editing the file busts the cache even though the filename argument did not change. Forget it and you get the classic confusion of a node that never re-runs after you change the file it reads. Second, node expansion, a node can return not just outputs but a request to splice a fresh subgraph into the running execution, which is how loops and recursion are built on top of a fundamentally acyclic graph. Third, lazy evaluation, a node can mark some inputs lazy and decide at runtime which it actually needs, so a switch node can avoid executing the branch it did not select, and an ExecutionBlocker can stop a branch from producing output at all. These features keep the graph model simple while allowing control flow that a pure DAG could not express.

Deep dive: the ModelPatcher and model management

The MODEL flowing on a wire is not the raw network, it is a ModelPatcher from comfy/model_patcher.py, and this indirection is what makes caching and LoRAs coexist. A ModelPatcher wraps a base model plus a set of patches, weight deltas and hooks, and its defining trick is clone, a cheap copy that shares the base weights but has its own patch list. When a LoraLoader node applies a LoRA it does not mutate the checkpoint, it clones the ModelPatcher and records the LoRA as patches on the clone, through comfy/sd.py's load_lora_for_models. The original checkpoint MODEL is untouched, so its cache entry stays valid and a second branch of the graph can use the unpatched model at the same time.

Load Checkpoint --> MODEL (ModelPatcher A, base weights W, no patches)
                        |
            LoraLoader clones A -> ModelPatcher B  (shares W, patch = LoRA delta)
                        |
   A still cached and usable         B is a distinct MODEL with its own signature
        |                                   |
   KSampler on A                       KSampler on B
   (base model)                        (LoRA-patched model)

Because patching produces a new object that shares weights rather than editing the old one, applying a LoRA is a constant-metadata operation that neither copies the model nor invalidates the base model's cache, and two differently patched versions of the same checkpoint can live side by side in one graph. When a patched model is finally about to run, comfy/model_management.py takes over. It estimates the model's memory, decides which models to keep resident and which to offload to system RAM, and materializes the patched weights on the GPU just in time, applying the patches during the load rather than storing a second full copy of the weights. The VRAM flags from Part II tune how aggressive that offloading is, and the smart-memory default tries to avoid reloading a model that is about to be used again. This is the layer that lets an SDXL model, a refiner, a ControlNet, and a VAE share a GPU that could not hold all of them at once.

Deep dive: the sampling pipeline as dataflow

The reason ComfyUI earns its complexity is that the sampling pipeline, the thing a one-click UI treats as a single opaque function, is here a set of typed values on wires. Walk what that buys, concretely. The sampler is a node, so you can chain two of them, a first low-resolution pass at full denoise and a second pass at partial denoise over an upscaled latent, which is the entire hires-fix technique expressed as an upscale node between two KSamplers with no special support in the engine. The conditioning is a value, so you can concatenate two encodings, restrict one to a region of the image, or run a ControlNet node that reads a pose or depth map and returns a modified CONDITIONING that steers the same sampler. The model is a value, so you can merge two checkpoints with a node, patch in a LoRA, or swap the sampler's guidance for a custom one. None of these require touching the engine because each is just a function whose input and output types line up.

Under the hood comfy/samplers.py separates the pieces so they can be recombined. The schedulers, normal, karras, exponential, and others, only produce a sequence of sigmas. The samplers, the euler and dpmpp families adapted from k-diffusion, only know how to step given a denoise function and those sigmas. The guidance, the classifier-free combination of conditional and unconditional predictions, lives in sampling_function and the CFGGuider, which is what advanced custom nodes subclass to change how the model's predictions are combined. The model itself is wrapped so the sampler sees a clean denoiser regardless of whether the underlying network predicts noise or velocity. Because these are orthogonal, a custom node can supply its own sigmas, its own sampler, or its own guidance and drop into the same SamplerCustom machinery.

This is the honest answer to why power users prefer the graph. A one-click UI must decide, on your behalf, that generation is exactly encode, sample, decode, and it exposes the parameters of that fixed shape. The moment your idea does not fit that shape, a second sampler, a mask that changes mid-pipeline, a model swapped between passes, conditioning assembled from three sources, you are stuck waiting for the UI's author to add a checkbox. ComfyUI's bet is that the pipeline itself should be user-editable data, so a new technique is usually a new node or a new wiring rather than a new release. The attention kernels inside the denoiser that all of this runs on are the subject of the FlashAttention chapter, and the diffusion process the sampler integrates is derived on the diffusion page.

Part VI: Reading the repository

ComfyUI is small enough to read the parts that matter in an afternoon. The core engine is a handful of files at the top level plus the comfy/ library. All paths below are on recent master and some have moved as the project refactors, so treat them as roles to find rather than fixed coordinates.

Stage 0, orientation. Read the README.md, then main.py and comfy/cli_args.py. Questions to hold, what does main.py start besides the web server, where is the prompt worker launched, and which flags change memory behavior and the cache policy?

Stage 1, the node contract. Read nodes.py, not all of it, but CheckpointLoaderSimple, CLIPTextEncode, EmptyLatentImage, KSampler with common_ksampler, VAEDecode, and SaveImage, then the NODE_CLASS_MAPPINGS block at the bottom. Questions, what exactly does each node declare, in what order does CheckpointLoaderSimple return its three outputs, and what makes SaveImage an output node?

Stage 2, the server and the API. server.py, focusing on the /prompt, /object_info, /view, and /ws handlers. Questions, what does /object_info return and why does the frontend need it, and how does a WebSocket message get addressed to one browser tab?

Stage 3, the executor and the cache. execution.py for validate_prompt and PromptExecutor.execute, then comfy_execution/graph.py and comfy_execution/caching.py. Questions, how is the set of nodes to run derived from the output nodes, what goes into a cache key, and how do IS_CHANGED, node expansion, and lazy inputs each bend the pure-DAG model?

Stage 4, the diffusion core. The comfy/ package, one file per concern, sd.py for loading checkpoints, CLIP, and VAE, model_patcher.py for the clone-and-patch model, model_management.py for VRAM and offloading, sample.py and samplers.py for noise, sigmas, samplers, and CFG, and model_base.py with ldm/ for the network definitions. Questions, why is a MODEL a ModelPatcher, where does classifier-free guidance actually compute, and how is a scheduler kept independent of a sampler?

Stage 5, the ecosystem and the frontier. folder_paths.py and extra_model_paths.yaml for where models live, comfy_extras/ for the extra built-in nodes, the custom_nodes/ loading path for how third-party packs register, and the separate ComfyUI_frontend repository for the editor. Questions, what is the minimum a custom-node package must export, and how does a workflow file differ from the API prompt?

Where not to start, the ldm/ model implementations are deep and model-specific and read better once you understand what calls them, the newer typed node schema is an in-flight change best met after the classic one is solid, and the frontend is a large TypeScript application whose internals are a separate study from the engine this chapter is about.

Part VII: Hands-on labs

All labs need a running server, python main.py, and one checkpoint in models/checkpoints/. Labs 1 through 4 need only the browser, labs 5 and 6 add a little Python. Log and message shapes vary with the fast pace of master.

Lab 1: watch one generation over the WebSocket. Concept, the life of a prompt from Part IV.

# open the browser dev tools, Network tab, filter to WS, then Queue Prompt
# you will see a stream of messages on /ws:
#   status -> execution_start -> execution_cached (node ids)
#   -> executing (per node) -> progress (per sampler step) -> executed -> done

Match each message to a stage of Part IV. Note that on the very first queue execution_cached is empty, every node misses.

Lab 2: prove the cache. Concept, input-signature keying.

# queue the default graph once (slow: model loads, sampler runs)
# edit only the positive prompt text, queue again
# in the WS stream, execution_cached now lists the checkpoint loader,
# the negative encoder, and the empty latent; only the positive encoder
# and everything downstream re-run

Then change the seed instead of the text and observe that the encoders stay cached while the sampler re-runs, the seed feeds only the sampler's signature. This is the whole caching model, made visible in two edits.

Lab 3: write and load a node. Concept, nodes as typed functions.

mkdir -p custom_nodes/my_pack
# paste the InvertImage class from Part II into custom_nodes/my_pack/__init__.py
# restart: python main.py

Find Invert Image under the image/postprocess category, wire it between VAE Decode and Save Image, and watch the colors flip. Then break it on purpose, change return (out,) to return out, restart, and read the error, the executor tried to unpack your tensor into multiple outputs.

Lab 4: hires fix as pure dataflow. Concept, the sampling pipeline is editable.

Empty Latent (512x512) -> KSampler #1 (denoise 1.0) -> Latent Upscale (1024x1024)
   -> KSampler #2 (same model, positive, negative, denoise ~0.5) -> VAE Decode -> Save

Build that graph. The second sampler starts from the upscaled latent and only partially denoises, sharpening detail. Nothing in the engine knows what hires fix is, you assembled it from an upscale node and a second sampler because the LATENT type let you. Compare the result to a single 1024 pass.

Lab 5: a LoRA leaves the checkpoint cached. Concept, the ModelPatcher.

# drop a LoRA into models/loras/, then in the graph:
#   Load Checkpoint -> Load LoRA (model, clip) -> CLIP Text Encode / KSampler
# queue once, then bypass the LoRA node and queue again

Watching the WebSocket, the checkpoint loader stays cached across both runs even though the model reaching the sampler changed. The LoRA loader cloned the ModelPatcher rather than editing the cached checkpoint, so the base model's signature never moved.

Lab 6: drive it headless. Concept, the prompt is the API.

import json, urllib.request, time

def submit(prompt):
    body = json.dumps({"prompt": prompt}).encode()
    r = urllib.request.urlopen(
        urllib.request.Request("http://127.0.0.1:8188/prompt", data=body))
    return json.loads(r.read())["prompt_id"]

def history(pid):
    r = urllib.request.urlopen(f"http://127.0.0.1:8188/history/{pid}")
    return json.loads(r.read())

# reuse the prompt dict from Part II, then:
pid = submit(prompt)
time.sleep(5)
print(history(pid))   # the recorded outputs, including saved filenames

Change one field of the prompt dict in a loop, a sweep over cfg or seed, and submit each. This is how ComfyUI becomes a batch backend, the graph is data, so a script can generate a hundred variations without a human touching the editor.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is ComfyUI, in one sentence?

A dataflow engine for diffusion pipelines whose nodes are typed pure functions over data types like MODEL, CONDITIONING, and LATENT, and whose executor runs the graph back to front from its output nodes while caching each node's output on the signature of its inputs.

2. What defines a node?

A Python class with an INPUT_TYPES classmethod, a RETURN_TYPES tuple, a FUNCTION naming the method to call, and a CATEGORY. The method takes the declared inputs as keyword arguments and returns a tuple matching RETURN_TYPES. Registration is adding the class to NODE_CLASS_MAPPINGS.

3. Why is the return a tuple even for one output?

Because RETURN_TYPES is a tuple of output slots and the executor pairs the returned tuple against it positionally. Returning a bare value that happens to be iterable, like a tensor, makes the executor try to unpack it across multiple slots, which is the most common node bug.

4. How does the cache decide what to recompute?

Each node's cache key is a signature built from its class and its fully resolved inputs, and a linked input resolves to the upstream node's own signature, so the key encodes the whole ancestor subgraph. Changing a value changes its node's signature and every descendant's, and leaves all unrelated signatures untouched, so exactly the affected nodes re-run.

5. When must a node implement IS_CHANGED, and why?

When its output depends on state the executor cannot see through its declared inputs, a file on disk, a clock, a network resource. IS_CHANGED returns a value folded into the signature, so changing the hidden state busts the cache. Without it a node that reads a file will serve the first result forever.

6. What is CONDITIONING, concretely?

A list of [embedding_tensor, extras_dict] pairs, the text encoder's output plus a dict later nodes use to attach area masks, strengths, or ControlNet hints. Because it is an ordinary value on a wire, conditioning can be combined, restricted to a region, or transformed by a node before the sampler sees it.

7. Why is a MODEL a ModelPatcher rather than a raw network?

So that patches like LoRAs can be applied by cloning the wrapper and recording deltas that share the base weights, instead of mutating the model. The unpatched checkpoint's cache entry stays valid, two differently patched versions can coexist in one graph, and the weights are only materialized with patches applied at load time.

8. Trace one text-to-image generation end to end.

The browser serializes the graph to an API prompt and POSTs /prompt, the server validates and enqueues it, the worker hands it to the executor, which plans the run from the output nodes and checks the cache per node, executes the misses, encoders then the sampler's denoising loop under CFG then the VAE decode, and the save node writes a self-describing PNG and streams the result back over the WebSocket.

9. Why does exposing the sampler as a node beat a one-click UI?

Because a fixed UI can only expose the parameters of the one pipeline shape its author chose, encode, sample, decode. When your idea needs a second sampler, a mid-pipeline mask, a model swap between passes, or conditioning assembled from several sources, the graph lets you wire it as long as the types match, with no engine change, while the form makes you wait for a new checkbox.

10. What is the difference between a workflow file and an API prompt?

A workflow is the editor's full document, node positions and link geometry included, meant for the browser. An API prompt is the compact form the server accepts, each node id mapped to its class and its inputs, where an input is a literal or a [source_id, slot] link. The engine runs the prompt, not the workflow.

11. When would you pick AUTOMATIC1111, Fooocus, or diffusers instead?

AUTOMATIC1111 or Forge for a tabbed form with a mature extension library and no graph to wire. Fooocus for a genuinely one-click, opinionated experience, noting it runs a Comfy-derived engine inside. Hugging Face diffusers when your output is a program or a service rather than an interactive graph. ComfyUI wins when you want explicit, reproducible, multi-stage control.

12. What is the real security risk of sharing workflows?

A workflow may reference custom nodes, and installing a custom-node pack drops Python into custom_nodes/ that the server imports at startup with your privileges. So a shared workflow can pressure you into running unaudited code. Treat custom nodes like any dependency and be wary of unfamiliar ones a workflow insists on.

13. How does ComfyUI fit an SDXL model, a ControlNet, and a VAE on a GPU that cannot hold all three?

model_management.py estimates each model's memory and offloads models to system RAM when they are not the one about to run, streaming the needed weights to the GPU just in time and honoring the VRAM flags. Smart memory avoids reloading a model that will be used again shortly.

14. How does control flow like loops exist on an acyclic graph?

A node can return a request to expand a fresh subgraph into the running execution, which builds loops and recursion on top of the DAG, and lazy inputs plus ExecutionBlocker let a node choose which branch to execute or stop a branch from producing output, so conditional flow works without cycles in the base model.

Part IX: Design lessons

Make the plugin contract a class shape, not an API. A node is four class attributes and a method, registered by adding it to a dict. There is almost nothing to learn and nothing to break between versions, which is exactly why thousands of custom nodes exist. When you want an ecosystem, minimize the surface a contributor must implement.

Cache on a signature of the inputs, not a timestamp. By keying each output on the recursive signature of everything that fed it, ComfyUI invalidates precisely the affected nodes and nothing else, with no manual dependency tracking. Content-addressed keys, build systems that hash inputs, and memoization all win the same way, derive the key from what the result actually depends on.

Wrap mutation as clone-and-patch. The ModelPatcher applies LoRAs by cloning a wrapper that shares weights rather than editing the model, so the cache stays valid and variants coexist. Persistent data structures, copy-on-write, and immutable configs with overlays are the same instinct, never mutate a shared thing when you can layer a cheap copy on top of it.

Expose the pipeline as data, not as a function. Turning encode, sample, and decode into typed nodes on wires means a new technique is usually new wiring rather than a new release. The general lesson is that user-editable dataflow beats a fixed form wherever users' needs outrun what any author can anticipate, the same reason spreadsheets and shader graphs and query planners endure.

Let the schema drive the UI. The frontend draws every node and validates every wire from the /object_info response, knowing nothing hard-coded about diffusion. A self-describing backend keeps the client generic and lets new nodes appear without a frontend change. Publish your schema and the tooling writes itself.

Keep the engine acyclic and add control flow above it. The base model is a pure DAG that is easy to cache and order, and loops, laziness, and branch-blocking are layered on as expansions rather than baked into the core. A simple, analyzable substrate with escape hatches beats a powerful but unanalyzable one.

Part X: Memorization framework

The one-sentence summary. ComfyUI serves a browser node editor whose graph is posted as an API prompt, validated, and enqueued, then an executor plans the run back to front from the output nodes, skips every node whose input signature is already cached, runs the rest, encoders then a CFG-guided sampling loop then a VAE decode, and saves a PNG that embeds the graph that made it.

browser -> POST /prompt -> validate_prompt -> PromptQueue
  -> prompt_worker -> PromptExecutor.execute
  -> ExecutionList (topo order from OUTPUT_NODEs)
  -> per node: cache hit on input signature? reuse : run NODE.FUNCTION
  -> KSampler (sigmas + sampler + CFG) -> VAE Decode -> Save Image
  -> result + progress streamed over /ws

The chain mapped to source.

launch          main.py, comfy/cli_args.py
server + API    server.py  (/prompt, /object_info, /view, /ws)
nodes           nodes.py  (NODE_CLASS_MAPPINGS), comfy_extras/
executor        execution.py, comfy_execution/{graph,caching}.py
model + patch   comfy/sd.py, comfy/model_patcher.py, comfy/model_management.py
sampling        comfy/sample.py, comfy/samplers.py, comfy/model_base.py, comfy/ldm/
paths + models  folder_paths.py, extra_model_paths.yaml
custom nodes    custom_nodes/*/__init__.py (NODE_CLASS_MAPPINGS, WEB_DIRECTORY)

Memorize these blocks.

• Node contract: INPUT_TYPES, RETURN_TYPES, FUNCTION, CATEGORY, and a method returning a tuple. Types are strings, wires are legal on type equality, combos are a list of options.
• Two ideas: nodes as typed functions, and output caching keyed on the recursive signature of a node's inputs.
• Execution: plan from OUTPUT_NODEs, run only what an output needs, reuse cache hits, invalidate exactly the descendants of a change.
• ModelPatcher: MODEL is a clone-and-patch wrapper sharing weights, so LoRAs do not mutate or invalidate the cached checkpoint.
• Data types: IMAGE is [B,H,W,C] float in [0,1], LATENT is {"samples": tensor}, CONDITIONING is a list of [embedding, extras] pairs.
• Formats: a workflow is the editor document, an API prompt is {id: {class_type, inputs}} with links [source_id, slot]. The engine runs the prompt.

Part XI: Papers and further reading

The ideas flowing on these wires come from a small set of papers, and each one rewards a direct read. Where this site derives the same idea in depth, the companion link points there.

1. Ho et al., Denoising Diffusion Probabilistic Models, 2020. The training objective behind every checkpoint these graphs load, derived step by step on the diffusion page of this site.
2. Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022. The Stable Diffusion paper, and the reason the sampler works on a LATENT that a VAE must decode. The diffusers walkthrough covers the same pipeline as a Python library rather than a graph.
3. Kingma and Welling, Auto-Encoding Variational Bayes, 2013. The autoencoder that maps between pixels and the latent grid, built from scratch on the VAE page.
4. Radford et al., Learning Transferable Visual Models From Natural Language Supervision, 2021. The CLIP text encoder that turns a prompt into CONDITIONING.
5. Ho and Salimans, Classifier-Free Diffusion Guidance, 2022. The cond and uncond combination that sampling_function computes at every step, with cfg as its scale.
6. Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022. The source of the karras sigmas and of the clean separation between schedulers and samplers that samplers.py mirrors, popularized by the k-diffusion library.
7. Lu et al., DPM-Solver++, Fast Solver for Guided Sampling of Diffusion Probabilistic Models, 2022. The solver family behind the dpmpp names in the sampler dropdown.
8. Podell et al., SDXL, Improving Latent Diffusion Models for High-Resolution Image Synthesis, 2023. The checkpoint this chapter's specimen run loads, placed in context by the diffusion and large vision models class.
9. Zhang et al., Adding Conditional Control to Text-to-Image Diffusion Models, ICCV 2023. The ControlNet paper, whose pose and depth hints ride in the CONDITIONING extras dict.
10. Hu et al., LoRA, Low-Rank Adaptation of Large Language Models, 2021. The low-rank deltas the ModelPatcher records as patches, covered as a library in the PEFT walkthrough.
11. Ye et al., IP-Adapter, Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models, 2023. The image-prompt adapter behind the IP-Adapter node packs, built on the mechanism of the cross-attention page.

Part XII: Final takeaway

If the diffusion machinery underneath is the gap, the ML implementations section builds the pieces from scratch, the diffusion, U-Net, and VAE pages cover exactly the model, denoiser, and decoder that flow on ComfyUI's wires, and the diffusion and large vision models class puts them in context. Then come back and read execution.py once more, it will read like a small, careful cache in front of a topological sort, which is exactly the point.