pixano_inference.models.vllm
Inference models for vLLM.
VLLMModel(name, vllm_model, model_config, processor_config, device=None)
Bases: BaseInferenceModel
Inference model for vLLM.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name
|
str
|
Name of the model. |
required |
vllm_model
|
str
|
The model Hugging Face hub's identifier. |
required |
model_config
|
dict[str, Any]
|
Configuration for the model. |
required |
processor_config
|
dict[str, Any]
|
Configuration for the processor of the model. |
required |
device
|
device | str | None
|
The device to use for inference. |
None
|
Source code in pixano_inference/models/vllm.py
metadata
property
Return the metadata of the model.
delete()
text_image_conditional_generation(prompt, temperature=1.0, max_new_tokens=16, **kwargs)
Generate text from an image and a prompt from the vLLM's LLM.chat
method.
Parameters: