Essential for many industries ranging from Hollywood computer-generated imagery to product design, 3D modelling tools often use text or image prompts to dictate different aspects of visual appearance, like colour and form.

Limited in realism

As much as this makes sense as a first point of contact, these systems are still limited in their realism due to their neglect of something central to the human experience: touch. 

PhD student Faraz Faruqi, lead author of a new paper on the project, says that TactStyle could have far-reaching applications extending from home decor and personal accessories to tactile learning tools. Photo: Mike Grimmett/MIT CSAIL.

Fundamental to the uniqueness of physical objects are their tactile properties, such as roughness, bumpiness, or the feel of materials like wood or stone.

Existing modelling methods often require advanced computer-aided design expertise and rarely support tactile feedback that can be crucial for how we perceive and interact with the physical world.

With that in mind, researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have created a new system for stylising 3D models using image prompts, effectively replicating both visual appearance and tactile properties.

Expected tactile properties

The CSAIL team’s 'TactStyle' tool allows creators to stylise 3D models based on images while also incorporating the expected tactile properties of the textures. TactStyle separates visual and geometric stylisation, enabling the replication of both visual and tactile properties from a single image input.

PhD student Faraz Faruqi, lead author of a paper on the project, says that TactStyle could have far-reaching applications, extending from home decor and personal accessories to tactile learning tools.

TactStyle enables users to download a base design – such as a headphone stand from Thingiverse – and customise it with the styles and textures they desire.

In education, learners can explore diverse textures from around the world without leaving the classroom, while in product design, rapid prototyping becomes easier as designers quickly print multiple iterations to refine tactile qualities.

“You could imagine using this sort of system for common objects, such as phone stands and earbud cases, to enable more complex textures and enhance tactile feedback in a variety of ways,” says Faruqi, who co-wrote the paper alongside MIT Associate Professor Stefanie Mueller, leader of the Human-Computer Interaction (HCI) Engineering Group at CSAIL.

“You can create tactile educational tools to demonstrate a range of different concepts in fields such as biology, geometry, and topography.”

Specialised tactile sensors

Traditional methods for replicating textures involve using specialised tactile sensors – such as GelSight, developed at MIT – that physically touch an object to capture its surface microgeometry as a 'height field'. But this requires having a physical object or its recorded surface for replication. TactStyle allows users to replicate the surface microgeometry by leveraging generative AI to generate a height field directly from an image of the texture.

On top of that, for platforms like the 3D printing repository Thingiverse, it is difficult to take individual designs and customise them. Indeed, if a user lacks sufficient technical background, changing a design manually runs the risk of actually 'breaking' it so that it cannot be printed any more.

All of these factors spurred Faruqi to wonder about building a tool that enables customisation of downloadable models on a high level, but that also preserves functionality.

In experiments, TactStyle showed significant improvements over traditional stylisation methods by generating accurate correlations between a texture’s visual image and its height field. This enables the replication of tactile properties directly from an image.

One psychophysical experiment showed that users perceive TactStyle’s generated textures as similar to both the expected tactile properties from visual input and the tactile features of the original texture, leading to a unified tactile and visual experience.

TactStyle leverages a pre-existing method, called 'Style2Fab', to modify the model’s colour channels to match the input image’s visual style.

Fine-tuned variational autoencoder

Users first provide an image of the desired texture, and then a fine-tuned variational autoencoder is used to translate the input image into a corresponding height field. This height field is then applied to modify the model’s geometry to create the tactile properties.

The colour and geometry stylisation modules work in tandem, stylising both the visual and tactile properties of the 3D model from a single image input.

Faruqi says that the core innovation lies in the geometry stylisation module, which uses a fine-tuned diffusion model to generate height fields from texture images – something previous stylisation frameworks do not accurately replicate.

Looking ahead, Faruqi says the team aims to extend TactStyle to generate novel 3D models using generative AI with embedded textures. This requires exploring exactly the sort of pipeline needed to replicate both the form and function of the 3D models being fabricated.

They also plan to investigate 'visuo-haptic mismatches' to create novel experiences with materials that defy conventional expectations, like something that appears to be made of marble but feels like it is made of wood.