Skip to content

base

Abstract interface for a pluggable render backend.

The Simulator constructs exactly one RenderBackend instance (exposed as og.sim.render_backend), chosen via gm.RENDER_BACKEND. This is the sibling abstraction to physics_backends.PhysicsBackend.

The interface covers application and Simulator-setup lifecycle, the render-graph "tick" primitive, and camera/modality capture. Viewport/GUI management and material authoring are still only partly relocated here; the remaining Kit-specific call sites are being moved behind this interface incrementally.

RenderBackend

Bases: ABC

Abstract base class for a render backend.

Source code in OmniGibson/omnigibson/render_backends/base.py 
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
class RenderBackend(ABC):
    """
    Abstract base class for a render backend.
    """

    def __init__(self, sim=None):
        self.sim = sim

    # ---- Application lifecycle ----

    def enable_extensions(self):
        """
        One-time, process-level setup performed once before any Simulator/Kit app exists.
        """

    def before_play(self, sim):
        """
        Hook called every time `Simulator.play()` transitions from stopped to playing.
        """

    # ---- Simulator setup hooks ----

    def apply_renderer_settings(self):
        """
        Applies this backend's renderer/lighting settings. Called once from `Simulator.__init__`.
        """

    def create_fabric_hierarchy(self):
        """
        Creates the Fabric/usdrt stage mirror and hierarchy interface this backend reads poses
        through, if it has one.

        Returns:
            2-tuple: (usdrt_stage, fabric_hierarchy), both None if this backend has no Fabric mirror
                (pose reads then go through raw pxr / the physics backend's own state instead).
        """
        return None, None

    def setup_viewer_camera(self, viewer_width, viewer_height):
        """
        Creates the interactive viewer camera at the given resolution, if this backend has a viewport.
        """

    # ---- Stage / application services ----

    @abstractmethod
    def create_default_pbr_material(self, scope_path, material_path, target_prim_path):
        """
        Creates a default PBR material at @material_path (under a Scope authored at @scope_path) and
        binds it to @target_prim_path.
        """

    @abstractmethod
    def get_stage(self):
        """
        Returns:
            pxr.Usd.Stage: The stage this backend renders from.
        """

    @abstractmethod
    def copy_prim(self, source_prim_path, dest_prim_path):
        """Copies the prim at @source_prim_path, and its whole subtree, to @dest_prim_path."""

    @abstractmethod
    def copy_mesh_prim(self, source_prim_path, dest_prim_path):
        """
        Copies the single, childless Mesh prim at @source_prim_path to @dest_prim_path. Narrower than
        copy_prim(); the caller guarantees there is no subtree to carry along.
        """

    @abstractmethod
    def deactivate_prim(self, prim):
        """Removes @prim from the scene as far as anything reading the stage is concerned."""

    @abstractmethod
    def delete_prim(self, prim_path, destructive=True):
        """
        Deletes the prim at @prim_path. @destructive=False deactivates it instead of removing it,
        which is all that is possible for a prim whose definition comes from a reference arc.
        """

    @abstractmethod
    def is_prim_ancestral(self, prim):
        """
        Returns:
            bool: Whether @prim's own definition was introduced via a reference arc (e.g. a link
                inside a dataset object's referenced USD), as opposed to being authored directly on
                the live stage. Such a prim can only be deactivated, never removed.
        """

    @abstractmethod
    def add_reference_to_stage(self, asset_path, prim_path):
        """
        References the USD at @asset_path in at @prim_path.

        Returns:
            Usd.Prim: The referencing prim.
        """

    @abstractmethod
    def create_primitive_mesh(self, primitive_type, prim_path, u_patches=None, v_patches=None, stage=None):
        """
        Authors a @primitive_type mesh (one of `PRIMITIVE_MESH_TYPES`) at @prim_path, optionally
        tessellated to @u_patches x @v_patches.
        """

    @abstractmethod
    def compute_world_aabb(self, prim_path):
        """
        Returns:
            2-tuple: (aabb_min, aabb_max) world-frame axis-aligned bounds of @prim_path, each a
                3-tuple of floats.
        """

    @abstractmethod
    def recompute_extents(self, prim):
        """Re-syncs @prim's authored extent attribute to its actual geometry."""

    @abstractmethod
    def add_semantic_labels(self, prim, label, instance_name="class"):
        """Tags @prim with semantic @label under @instance_name, for segmentation rendering."""

    @abstractmethod
    def suppress_log(self, channels=None):
        """
        Returns:
            contextmanager: Suppresses @channels (or all logging, if None) for its duration.
        """

    # ---- Render-graph lifecycle ----

    @abstractmethod
    def render(self):
        """
        Advance/tick the render graph by one frame (a Kit app-update + RTX draw, or equivalent). Does
        NOT itself return pixel data -- that is a later phase's `capture_modality()`-style method.
        """

    # ---- Camera capture pipeline ----

    @abstractmethod
    def create_camera_resource(self, prim_path, resolution, force_new=False):
        """
        Creates a backend-specific rendering resource (e.g. a Replicator render product) for the camera
        prim at @prim_path, sized to @resolution = (width, height). @force_new forces creation of a new
        resource even if the backend has a stale cached one at the same @prim_path (needed when
        recreating a resource right after destroying the previous one at the same path).

        Returns:
            object: Opaque handle to be passed to attach_modality/destroy_camera_resource.
        """

    @abstractmethod
    def destroy_camera_resource(self, camera_resource):
        """Destroys a camera resource previously returned by create_camera_resource."""

    @abstractmethod
    def create_annotator(self, raw_modality_name):
        """
        Creates (but does not yet attach) a backend-specific annotator for @raw_modality_name (one of
        VisionSensor.RAW_SENSOR_TYPES' values).

        Returns:
            object: Opaque handle to be passed to attach_modality/detach_modality/get_modality_data.
        """

    @abstractmethod
    def attach_modality(self, annotator, camera_resource):
        """Attaches @annotator (from create_annotator) to @camera_resource so it starts collecting data."""

    @abstractmethod
    def detach_modality(self, annotator, camera_resource):
        """Detaches @annotator from @camera_resource. Tolerates backend-specific teardown quirks."""

    @abstractmethod
    def get_modality_data(self, annotator, device=None):
        """
        Returns:
            dict or array: The current raw observation for @annotator -- either a dict with "data"/"info"
                keys or a direct array/tensor, depending on modality (mirrors the underlying annotator's
                own return convention). @device, if given, requests data placed on that device.
        """

add_reference_to_stage(asset_path, prim_path) abstractmethod

References the USD at @asset_path in at @prim_path.

Returns:

Type Description
Prim

The referencing prim.
Source code in OmniGibson/omnigibson/render_backends/base.py 
106
107
108
109
110
111
112
113
@abstractmethod
def add_reference_to_stage(self, asset_path, prim_path):
    """
    References the USD at @asset_path in at @prim_path.

    Returns:
        Usd.Prim: The referencing prim.
    """

add_semantic_labels(prim, label, instance_name='class') abstractmethod

Tags @prim with semantic @label under @instance_name, for segmentation rendering.

Source code in OmniGibson/omnigibson/render_backends/base.py 
134
135
136
@abstractmethod
def add_semantic_labels(self, prim, label, instance_name="class"):
    """Tags @prim with semantic @label under @instance_name, for segmentation rendering."""

apply_renderer_settings()

Applies this backend's renderer/lighting settings. Called once from Simulator.__init__.

Source code in OmniGibson/omnigibson/render_backends/base.py 
38
39
40
41
def apply_renderer_settings(self):
    """
    Applies this backend's renderer/lighting settings. Called once from `Simulator.__init__`.
    """

attach_modality(annotator, camera_resource) abstractmethod

Attaches @annotator (from create_annotator) to @camera_resource so it starts collecting data.

Source code in OmniGibson/omnigibson/render_backends/base.py 
182
183
184
@abstractmethod
def attach_modality(self, annotator, camera_resource):
    """Attaches @annotator (from create_annotator) to @camera_resource so it starts collecting data."""

before_play(sim)

Hook called every time Simulator.play() transitions from stopped to playing.

Source code in OmniGibson/omnigibson/render_backends/base.py 
31
32
33
34
def before_play(self, sim):
    """
    Hook called every time `Simulator.play()` transitions from stopped to playing.
    """

compute_world_aabb(prim_path) abstractmethod

Returns:

Type Description
2 - tuple

(aabb_min, aabb_max) world-frame axis-aligned bounds of @prim_path, each a 3-tuple of floats.
Source code in OmniGibson/omnigibson/render_backends/base.py 
122
123
124
125
126
127
128
@abstractmethod
def compute_world_aabb(self, prim_path):
    """
    Returns:
        2-tuple: (aabb_min, aabb_max) world-frame axis-aligned bounds of @prim_path, each a
            3-tuple of floats.
    """

copy_mesh_prim(source_prim_path, dest_prim_path) abstractmethod

Copies the single, childless Mesh prim at @source_prim_path to @dest_prim_path. Narrower than copy_prim(); the caller guarantees there is no subtree to carry along.

Source code in OmniGibson/omnigibson/render_backends/base.py 
79
80
81
82
83
84
@abstractmethod
def copy_mesh_prim(self, source_prim_path, dest_prim_path):
    """
    Copies the single, childless Mesh prim at @source_prim_path to @dest_prim_path. Narrower than
    copy_prim(); the caller guarantees there is no subtree to carry along.
    """

copy_prim(source_prim_path, dest_prim_path) abstractmethod

Copies the prim at @source_prim_path, and its whole subtree, to @dest_prim_path.

Source code in OmniGibson/omnigibson/render_backends/base.py 
75
76
77
@abstractmethod
def copy_prim(self, source_prim_path, dest_prim_path):
    """Copies the prim at @source_prim_path, and its whole subtree, to @dest_prim_path."""

create_annotator(raw_modality_name) abstractmethod

Creates (but does not yet attach) a backend-specific annotator for @raw_modality_name (one of VisionSensor.RAW_SENSOR_TYPES' values).

Returns:

Type Description
object

Opaque handle to be passed to attach_modality/detach_modality/get_modality_data.
Source code in OmniGibson/omnigibson/render_backends/base.py 
172
173
174
175
176
177
178
179
180
@abstractmethod
def create_annotator(self, raw_modality_name):
    """
    Creates (but does not yet attach) a backend-specific annotator for @raw_modality_name (one of
    VisionSensor.RAW_SENSOR_TYPES' values).

    Returns:
        object: Opaque handle to be passed to attach_modality/detach_modality/get_modality_data.
    """

create_camera_resource(prim_path, resolution, force_new=False) abstractmethod

Creates a backend-specific rendering resource (e.g. a Replicator render product) for the camera prim at @prim_path, sized to @resolution = (width, height). @force_new forces creation of a new resource even if the backend has a stale cached one at the same @prim_path (needed when recreating a resource right after destroying the previous one at the same path).

Returns:

Type Description
object

Opaque handle to be passed to attach_modality/destroy_camera_resource.
Source code in OmniGibson/omnigibson/render_backends/base.py 
156
157
158
159
160
161
162
163
164
165
166
@abstractmethod
def create_camera_resource(self, prim_path, resolution, force_new=False):
    """
    Creates a backend-specific rendering resource (e.g. a Replicator render product) for the camera
    prim at @prim_path, sized to @resolution = (width, height). @force_new forces creation of a new
    resource even if the backend has a stale cached one at the same @prim_path (needed when
    recreating a resource right after destroying the previous one at the same path).

    Returns:
        object: Opaque handle to be passed to attach_modality/destroy_camera_resource.
    """

create_default_pbr_material(scope_path, material_path, target_prim_path) abstractmethod

Creates a default PBR material at @material_path (under a Scope authored at @scope_path) and binds it to @target_prim_path.

Source code in OmniGibson/omnigibson/render_backends/base.py 
61
62
63
64
65
66
@abstractmethod
def create_default_pbr_material(self, scope_path, material_path, target_prim_path):
    """
    Creates a default PBR material at @material_path (under a Scope authored at @scope_path) and
    binds it to @target_prim_path.
    """

create_fabric_hierarchy()

Creates the Fabric/usdrt stage mirror and hierarchy interface this backend reads poses through, if it has one.

Returns:

Type Description
2 - tuple

(usdrt_stage, fabric_hierarchy), both None if this backend has no Fabric mirror (pose reads then go through raw pxr / the physics backend's own state instead).
Source code in OmniGibson/omnigibson/render_backends/base.py 
43
44
45
46
47
48
49
50
51
52
def create_fabric_hierarchy(self):
    """
    Creates the Fabric/usdrt stage mirror and hierarchy interface this backend reads poses
    through, if it has one.

    Returns:
        2-tuple: (usdrt_stage, fabric_hierarchy), both None if this backend has no Fabric mirror
            (pose reads then go through raw pxr / the physics backend's own state instead).
    """
    return None, None

create_primitive_mesh(primitive_type, prim_path, u_patches=None, v_patches=None, stage=None) abstractmethod

Authors a @primitive_type mesh (one of PRIMITIVE_MESH_TYPES) at @prim_path, optionally tessellated to @u_patches x @v_patches.

Source code in OmniGibson/omnigibson/render_backends/base.py 
115
116
117
118
119
120
@abstractmethod
def create_primitive_mesh(self, primitive_type, prim_path, u_patches=None, v_patches=None, stage=None):
    """
    Authors a @primitive_type mesh (one of `PRIMITIVE_MESH_TYPES`) at @prim_path, optionally
    tessellated to @u_patches x @v_patches.
    """

deactivate_prim(prim) abstractmethod

Removes @prim from the scene as far as anything reading the stage is concerned.

Source code in OmniGibson/omnigibson/render_backends/base.py 
86
87
88
@abstractmethod
def deactivate_prim(self, prim):
    """Removes @prim from the scene as far as anything reading the stage is concerned."""

delete_prim(prim_path, destructive=True) abstractmethod

Deletes the prim at @prim_path. @destructive=False deactivates it instead of removing it, which is all that is possible for a prim whose definition comes from a reference arc.

Source code in OmniGibson/omnigibson/render_backends/base.py 
90
91
92
93
94
95
@abstractmethod
def delete_prim(self, prim_path, destructive=True):
    """
    Deletes the prim at @prim_path. @destructive=False deactivates it instead of removing it,
    which is all that is possible for a prim whose definition comes from a reference arc.
    """

destroy_camera_resource(camera_resource) abstractmethod

Destroys a camera resource previously returned by create_camera_resource.

Source code in OmniGibson/omnigibson/render_backends/base.py 
168
169
170
@abstractmethod
def destroy_camera_resource(self, camera_resource):
    """Destroys a camera resource previously returned by create_camera_resource."""

detach_modality(annotator, camera_resource) abstractmethod

Detaches @annotator from @camera_resource. Tolerates backend-specific teardown quirks.

Source code in OmniGibson/omnigibson/render_backends/base.py 
186
187
188
@abstractmethod
def detach_modality(self, annotator, camera_resource):
    """Detaches @annotator from @camera_resource. Tolerates backend-specific teardown quirks."""

enable_extensions()

One-time, process-level setup performed once before any Simulator/Kit app exists.

Source code in OmniGibson/omnigibson/render_backends/base.py 
26
27
28
29
def enable_extensions(self):
    """
    One-time, process-level setup performed once before any Simulator/Kit app exists.
    """

get_modality_data(annotator, device=None) abstractmethod

Returns:

Type Description
dict or array

The current raw observation for @annotator -- either a dict with "data"/"info" keys or a direct array/tensor, depending on modality (mirrors the underlying annotator's own return convention). @device, if given, requests data placed on that device.
Source code in OmniGibson/omnigibson/render_backends/base.py 
190
191
192
193
194
195
196
197
@abstractmethod
def get_modality_data(self, annotator, device=None):
    """
    Returns:
        dict or array: The current raw observation for @annotator -- either a dict with "data"/"info"
            keys or a direct array/tensor, depending on modality (mirrors the underlying annotator's
            own return convention). @device, if given, requests data placed on that device.
    """

get_stage() abstractmethod

Returns:

Type Description
Stage

The stage this backend renders from.
Source code in OmniGibson/omnigibson/render_backends/base.py 
68
69
70
71
72
73
@abstractmethod
def get_stage(self):
    """
    Returns:
        pxr.Usd.Stage: The stage this backend renders from.
    """

is_prim_ancestral(prim) abstractmethod

Returns:

Type Description
bool

Whether @prim's own definition was introduced via a reference arc (e.g. a link inside a dataset object's referenced USD), as opposed to being authored directly on the live stage. Such a prim can only be deactivated, never removed.
Source code in OmniGibson/omnigibson/render_backends/base.py 
 97
 98
 99
100
101
102
103
104
@abstractmethod
def is_prim_ancestral(self, prim):
    """
    Returns:
        bool: Whether @prim's own definition was introduced via a reference arc (e.g. a link
            inside a dataset object's referenced USD), as opposed to being authored directly on
            the live stage. Such a prim can only be deactivated, never removed.
    """

recompute_extents(prim) abstractmethod

Re-syncs @prim's authored extent attribute to its actual geometry.

Source code in OmniGibson/omnigibson/render_backends/base.py 
130
131
132
@abstractmethod
def recompute_extents(self, prim):
    """Re-syncs @prim's authored extent attribute to its actual geometry."""

render() abstractmethod

Advance/tick the render graph by one frame (a Kit app-update + RTX draw, or equivalent). Does NOT itself return pixel data -- that is a later phase's capture_modality()-style method.

Source code in OmniGibson/omnigibson/render_backends/base.py 
147
148
149
150
151
152
@abstractmethod
def render(self):
    """
    Advance/tick the render graph by one frame (a Kit app-update + RTX draw, or equivalent). Does
    NOT itself return pixel data -- that is a later phase's `capture_modality()`-style method.
    """

setup_viewer_camera(viewer_width, viewer_height)

Creates the interactive viewer camera at the given resolution, if this backend has a viewport.

Source code in OmniGibson/omnigibson/render_backends/base.py 
54
55
56
57
def setup_viewer_camera(self, viewer_width, viewer_height):
    """
    Creates the interactive viewer camera at the given resolution, if this backend has a viewport.
    """

suppress_log(channels=None) abstractmethod

Returns:

Type Description
contextmanager

Suppresses @channels (or all logging, if None) for its duration.
Source code in OmniGibson/omnigibson/render_backends/base.py 
138
139
140
141
142
143
@abstractmethod
def suppress_log(self, channels=None):
    """
    Returns:
        contextmanager: Suppresses @channels (or all logging, if None) for its duration.
    """