Skip to content

pixano_inference.models.vlm

VLM (Vision-Language Model) base class.

The I/O types live in :mod:pixano_inference.schemas.vlm and are re-exported here so from pixano_inference.models.vlm import VLMInput keeps working.

VLMModel(config)

Bases: InferenceModel[VLMInput, VLMOutput]

Base class for vision-language models.

predict receives a :class:VLMInput (prompt, images and generation parameters) and returns a :class:VLMOutput (generated text, usage and generation config). The interface property, when overridden, returns a :class:VLMInterface: the prompt forms the model accepts and how many images a request may carry.

Example 
@register_model("my-vlm")
class MyVLM(VLMModel):
    def load_model(self):
        self.model = load_weights(self.config.model_params["path"])

    def predict(self, input: VLMInput) -> VLMOutput:
        text = self.model.generate(input.prompt, input.images)
        return VLMOutput(generated_text=text, usage=..., generation_config=...)
Source code in pixano_inference/models/base.py 
53
54
55
56
57
58
59
def __init__(self, config: ModelDeploymentConfig) -> None:
    """Initialize the model with deployment config.

    Args:
        config: Model deployment configuration.
    """
    self._config = config