ControlNet

ControlNet API in the All-In-One solution is for specific customers only.

ControlNet enhances model customization by providing finer control over the generated outputs. It adds several parameters on top of the original API, allowing users to manipulate aspects such as proportions, shapes, and structures of 3D models.

For information on the basic API requests, please refer to the Base Rodin API Documentation.

Get started with Rodin

Additional ControlNet Parameters

ControlNet introduces three main parameters to provide advanced control over the model generation process:

  1. BoundingBox ControlNet

  2. Voxel ControlNet

  3. Point Cloud ControlNet

1. BoundingBox ControlNet

The BoundingBox ControlNet allows users to define the proportions of the generated model by specifying the length, width, and height through a draggable bounding box. This is particularly useful when you want the generated object to fit within specific dimensions or adhere to certain spatial constraints.

Use Case

  • Example: Generating a sofa or a treasure chest that must fit within prescribed dimensions.

Parameter Details

{
  "bbox_condition": [
    100,
    100,
    100
  ]
}

  • bbox_condition: An array that specifies the dimensions and scaling factor of the bounding box.

    • Elements:

      1. Length (X-axis): 100 units.

      2. Width (Y-axis): 100 units.

      3. Height (Z-axis): 100 units.

By setting the bbox_condition, you're instructing the model to generate an object that fits within a box of the specified dimensions. The scaling factor allows you to proportionally scale the bounding box dimensions if needed.

Example Usage

Here's how you can use the BoundingBox ControlNet parameter in a cURL request:

export RODIN_API_KEY="your_api_key"
curl https://hyperhuman.deemos.com/api/v2/rodin \
  -H "Authorization: Bearer ${RODIN_API_KEY}" \
  -F "images=@/path/to/your/image.jpg" \
  -F "bbox_condition=[100,100,100]"
unset RODIN_API_KEY

2. Voxel ControlNet

Voxel ControlNet enables users to create or convert an existing model into a voxel representation, which can then be used to guide the overall shape and proportions of the new model. This is especially useful for maintaining specific body proportions or creating model variants while preserving the original structures.

Use Case

  • Example: Generating a multi-head character (with exaggerated proportions) by using an existing multi-head model converted into voxels to guide the new model generation.

Parameter Details

{
  "voxel_condition": "<Base64 Encoded Data>",
  "voxel_condition_cfg": true,
  "voxel_condition_weight": 1
}

  • voxel_condition: A string containing Base64-encoded voxel data representing the 3D structure used as a condition for model generation(Here is an example of the conversion).

import numpy as np
import base64

voxel = np.random.choice([True, False], size=(16, 16, 16)) # fixed size.
voxel_condition = base64.b64encode(
    np.packbits(voxel, bitorder="little").tobytes()
).decode()
voxel_condition_parsed = np.unpackbits(
    np.frombuffer(base64.b64decode(voxel_condition.encode()), np.uint8), bitorder="little"
).astype(bool).reshape(16, 16, 16)

print(voxel_condition)
assert np.all(voxel == voxel_condition_parsed)

  • voxel_condition_cfg: A boolean value (true or false) that enables or disables the voxel condition guidance.

  • voxel_condition_weight: A numerical value that determines the influence weight of the voxel condition on the generation process.

    • A higher weight increases the influence of the voxel condition, making the generated model more closely adhere to the provided voxel structure.

By utilizing the voxel_condition parameters, you can exert precise control over the generated model's structural features. This is particularly effective for:

  • Maintaining stylistic proportions.

  • Generating variants of existing models.

  • Ensuring consistency across a series of generated models.

Example Usage

Assuming you have Base64-encoded voxel data stored in voxel_data.txt, here's how you can include it in your API request:

export RODIN_API_KEY="your_api_key"
export VOXEL_DATA=$(cat voxel_data.txt)

curl https://hyperhuman.deemos.com/api/v2/rodin \
  -H "Authorization: Bearer ${RODIN_API_KEY}" \
  -F "images=@/path/to/your/image.jpg" \
  -F "voxel_condition=${VOXEL_DATA}" \
  -F "voxel_condition_cfg=true" \
  -F "voxel_condition_weight=1"
unset RODIN_API_KEY
unset VOXEL_DATA

3. Point Cloud ControlNet

Point Cloud ControlNet allows users to create or use an existing model converted into a point cloud to control the surface details, proportions, and overall shape of the new model. This method is beneficial for refining models, repairing low-quality 3D scans, and generating variants that maintain specific surface characteristics.

Use Case

  • Example: Repairing a low-quality 3D scanned model by using its point cloud as a guide to generate a higher-quality version. Additionally, it can be combined with 2D image generation to create diverse variants of an existing model.

Parameter Details

{
  "pcd_condition": [
    [0.5206083885457375, 0.7898993724346586, -0.01808462914092718],
    ...,
    []
  ],
  "pcd_condition_weight": 0.01
}

  • pcd_condition: An array of arrays, each containing the X, Y, and Z coordinates of a point in the point cloud.

    • Each sub-array represents a point in 3D space that collectively defines the shape and surface characteristics of the model to be generated.

  • pcd_condition_weight: A numerical value representing the weight or tolerance level for the point cloud condition.

    • This parameter allows flexibility in how strictly the model should adhere to the provided point cloud. A lower value means the model will more closely follow the point cloud, while a higher value allows for more variation.

Using the pcd_condition parameters, you can guide the model generation process to produce outputs that conform to a specific surface structure and shape defined by a point cloud. This is useful for:

  • Enhancing or refining existing models.

  • Repairing defects in 3D scans.

  • Creating new models that share surface characteristics with a reference model.

Example Usage

Assuming you have your point cloud data in point_cloud.json, you can include it in your API request as follows:

export RODIN_API_KEY="your_api_key"
export PCD_CONDITION=$(cat point_cloud.json)

curl https://hyperhuman.deemos.com/api/v2/rodin \
  -H "Authorization: Bearer ${RODIN_API_KEY}" \
  -F "images=@/path/to/your/image.jpg" \
  -F "pcd_condition=${PCD_CONDITION}" \
  -F "pcd_condition_weight=0.01"
unset RODIN_API_KEY
unset PCD_CONDITION

Online Parameter Generator

To simplify the process of generating and configuring these parameters, we provide an online parameter generator tool:

LogoControlNet Playgroundhyper3d-controlnet-generator.pages.dev

Notes:

  • Replace <your_api_key> with your actual API key.

  • Only one ControlNet parameter is used in the request, as multiple ControlNets cannot be combined in a single request.

Last updated