Yaroslav Ganin

A new rendering model just appeared for full global illumination. It's ... a transformer! The start of Mesh-to-image, end-to-end physics based rendering? Let's have a look 👇👇👇 Zhejiang University, Microsoft Research Asia and College of William & Mary present "Transformer-based Neural Rendering of Triangle Meshes with Global Illumination". This paper will be presented at ACM SIGGRAPH 2025. RenderFormer is a neural rendering pipeline that directly renders an image from a triangle-based representation of a scene with full global illumination effects and that does not require per-scene training or fine-tuning. Instead of taking a physics-centric approach to rendering, they formulate rendering as a sequence-to-sequence transformation where a sequence of tokens representing triangles with reflectance properties is converted to a sequence of output tokens representing small patches of pixels. RenderFormer follows a two stage pipeline: a view-independent stage that models triangle-to-triangle light transport, and a view-dependent stage that transforms a token representing a bundle of rays to the corresponding pixel values guided by the triangle-sequence from the the view-independent stage. Both stages are based on the transformer architecture and are learned with minimal prior constraints. No rasterization, no ray tracing Amazing stuff, more info in the comments #globalillumination #neuralrendering #transformers #machinelearning #endtoend #deeplearning

335

11 Comments

Some recent, exciting news on compressed tokenizers and how individual tokens modify images in encoder-decoder models. I hope this opens up a new avenue of research with smaller models across frameworks (perhaps JEPA?). https://lnkd.in/evFw8WBu

69

2 Comments

Ever wondered how linear RNNs like #mLSTM (#xLSTM) or #Mamba can be extended to multiple dimensions? Check out "pLSTM: parallelizable Linear Source Transition Mark networks". #pLSTM works on sequences, images, (directed acyclic) graphs, you name it. Let's see if you can get the paper from the visual abstract only. 😉 Paper link: https://lnkd.in/ddgs78x6 Code: https://lnkd.in/dy7jHhRn

182

6 Comments

Mayank M. put together an excellent tutorial on integrating reference motion tracking into Isaac Lab using a simple extension template. GitHub: https://lnkd.in/d8sj7iwb ⭐ While Isaac Lab can be challenging to get started with, this guide makes it much more approachable. I highly recommend checking it out, and if you find it helpful, consider giving it a ⭐ on GitHub!

188

New paper alert! We present the world’s largest first-principle dataset for polyanionic sodium cathode materials, comprising nearly 300,000 DFT-calculated structures. Creating a huge leap forward in the search for promising cathode materials for sodium-ion batteries. This dataset includes optimizations of structures, ab initio molecular dynamics simulation trajectories sampled at 1000 K, and structures generated from ML-driven molecular dynamics simulation at 1000 K using an active learning algorithm. Moreover, this database contains both ordered structures and site-disordered structures, where two transition metal ion occupy the same symmetry site. The dataset provides key structural and electronic properties, including total energies, atomic forces, magnetic moments, and atomic charges, offering a detailed representation of the phase space relevant to sodium-ion cathodes. By extending beyond equilibrium structures, our dataset improves machine learning models for energy materials and enhances insights into ionic transport in polyanionic compounds. This dataset will benefit researchers in computational materials science, battery materials, and atomistic modeling. https://lnkd.in/dpyQzkNg

142

3 Comments

From experience came understanding. In a recent interview with Dwarkesh Patel, Richard Sutton — father of reinforcement learning and winner of the 2024 Turing Award — argues that large language models are a dead end. Why? Because they can’t learn while acting. They only replay what they’ve already digested. For Sutton, true intelligence doesn’t come from scaling static models, but from agents that continuously adapt — that improve through interaction with their environment. Learning, in this view, isn’t a phase of training; it’s a way of being. It’s the same spirit that shaped reinforcement learning decades ago: act, receive feedback, adjust. A loop simple enough to fit on a napkin — yet powerful enough to teach machines how to walk, play, trade, and win. Whether or not LLMs are a “dead end,” Sutton’s point is deeper: Progress in AI won’t come from memorizing more — it will come from experiencing more. From algorithms that live, not just compute. Watch the full interview: https://lnkd.in/dVv6vgXU #ArtificialIntelligence #ReinforcementLearning #MachineLearning #RichardSutton #TuringAward #AIPhilosophy #AIResearch #DeepLearning #ContinualLearning #IntelligentAgents #AIforScience #LearningSystems #CognitiveAI #FutureOfAI #AIandSociety

8