Trajectory value learning for autonomous systems includes generating an environment image from sensor input and processing the environment image through an image neural network to obtain a feature map. Trajectory value learning further includes sampling possible trajectories to obtain a candidate trajectory for an autonomous system, extracting, from the feature map, feature vectors corresponding to the candidate trajectory, combining the feature vectors into the input vector, and processing, by a score neural network model, the input vector to obtain a projected score for the candidate trajectory. Trajectory value learning further includes selecting, from the candidate trajectories, the candidate trajectory as a selected trajectory based on the projected score, and implementing the selected trajectory.
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
B60W 50/06 - Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
G05B 13/02 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
G06T 7/246 - Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
G06T 7/73 - Determining position or orientation of objects or cameras using feature-based methods
Learning to drive via asymmetric self-play includes executing a scenario that includes a set of actors. A student action in the scenario is determined using a student model for a student actor of the set of actors, and a teacher action in the scenario is determined using a teacher model for a teacher actor of the set of actors. Learning to drive further involves processing a student reward function based on the student action to reduce a student collision likelihood of the student model and processing a teacher reward function based on the teacher action to reduce a teacher collision likelihood of the teacher model and increase the student collision likelihood of the student model. The student model and the teacher model are iteratively updated using the student reward function and the teacher reward function. The student model is saved as a virtual driver of an autonomous system.
Gradient guided object reconstruction includes rendering a predicted image from a current object model of an object in a scenario. Multiple gradients are generated based on a comparison of the predicted image with a real image. The current object model is augmented with the gradients to generate a reconstruction input. A reconstruction network processes the reconstruction input to generate an updated object model as the current object model and the updated object model is stored.
A method implements generating and testing augmented data for an autonomous system. The method involves generating system data from multiple sensors of an autonomous system operating in a real-world environment. The method further involves generating simulation data that includes a perturbation to the system data. The method further involves augmenting the system data to include the perturbation of the simulation data and generate augmented data. The method further involves injecting the augmented data into one or more components of the autonomous system to test the autonomous system with the perturbation.
A method implements a road mapping framework. The method includes executing an extraction model to generate multiple lane features from a lane image. The method further includes executing a coarse model to generate multiple coarse boundary embeddings and a coarse lane graph from the lane features and multiple prior boundary embeddings using a transformer decoder. The prior boundary embeddings is generated from a prior lane graph. The method further includes executing a refinement model to update the prior lane graph with a refined lane graph to form an updated lane graph. The refined lane graph is generated from multiple refined boundary embeddings that is output from a transformer encoder. The transformer encoder generates the refined boundary embeddings from the coarse boundary embeddings combined with multiple point embeddings corresponding to the coarse boundary embeddings.
G06V 10/77 - Processing image or video features in feature spacesArrangements for image or video recognition or understanding using pattern recognition or machine learning using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]Blind source separation
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
A method implements a neural calibration framework. The method includes generating a simulated sensor rendering using a feature grid and calibration data. The method further includes generating a loss value between the simulated sensor rendering and a real-world sensor rendering. The method further includes updating the feature grid and the calibration data using the loss value to generate an updated feature grid and updated calibration data to reduce a subsequent loss value. The method further includes calibrating the sensor with the updated calibration data.
Unsupervised occupancy forecasting includes processing, by an implicit occupancy model, previous LiDAR points to create a feature map and extrapolating location points having a corresponding location identifier in the geographic region from future LiDAR points. The future LiDAR points each include a corresponding timestamp defined by a time of acquisition of the future LiDAR point. Unsupervised occupancy forecasting includes further includes sampling the location points to obtain a sample set of location points that includes a set of training query points with the corresponding timestamp and a corresponding set of actual point attributes, processing, by the implicit occupancy model using the feature map, the set of training query points to obtain a set of predicted point attributes for the corresponding timestamp, and updating the implicit occupancy model according to a comparison of the set of predicted point attributes and the corresponding set of actual point attributes.
The operations of autonomous system training and testing may include generating a digital twin of a real world scenario as a simulated environment state of a simulated environment. The operations may also include iteratively, through multiple timesteps: executing a sensor simulation model on the simulated environment state to obtain simulated sensor output, obtaining, from a virtual driver of an autonomous system, at least one actuation action that is based on the simulated sensor output, updating an autonomous system state of the autonomous system based on the at least one actuation action, modeling, using multiple actor models, multiple actors in the simulated environment according to the simulated environment state to obtain multiple actor actions, and updating the simulated environment state according to the actor actions and the autonomous system state. The operations may furthermore include evaluating the virtual driver after updating the simulated environment state.
A method learns unsupervised world models for autonomous driving via discrete diffusion. The method includes encoding an observation of an actor for a geographic region using an encoder to generate a prior frame of prior tokens. The method further includes processing the prior frame with a spatio-temporal transformer to generate a predicted frame of predicted tokens. The spatio-temporal transformer includes a spatial transformer and a temporal transformer. The method further includes processing the predicted frame to generate a predicted action for the actor. The method further includes decoding the predicted frame to generate a predicted observation of the geographic region.
G06F 30/27 - Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
B60W 50/00 - Details of control systems for road vehicle drive control not related to the control of a particular sub-unit
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
10.
AUTOMATIC LABELING OF OBJECTS FROM LIDAR POINT CLOUDS VIA TRAJECTORY-LEVEL REFINEMENT
A method implements automatic labeling of objects from LiDAR point clouds via trajectory level refinement. The method includes executing an encoder model using a set of bounding box vectors and a set of point clouds to generate a set of combined feature vectors and executing an attention model using the set of combined feature vectors to generate a set of updated feature vectors. The method further includes executing a decoder model using the set of updated feature vectors to generate a set of pose residuals and a size residual and updating the set of bounding box vectors with the set of pose residuals and the size residual to generate a set of refined bounding box vectors. The method further includes executing an action responsive to the set of refined bounding box vectors.
Latent representation based appearance modification for adversarial testing and training include obtaining a first latent representation of an actor, performing a modification of the first latent representation of an actor to obtain a second latent representation, and generating a 3D model from the second latent representation. The operations further include performing, by a simulator interacting with the virtual driver, a simulation of the virtual world having the 3D model of the actor and the autonomous system moving in the virtual world, evaluating the virtual driver interacting in the virtual world during the simulation to obtain an evaluation result, and outputting the evaluation result.
A method implements multimodal four-dimensional panoptic segmentation. The method includes receiving a set of images and a set of point clouds and executing an image encoder model using the set of images to extract a set of image feature maps. The method further includes executing a point voxel encoder model using the set of image feature maps and the set of point clouds to extract a set of voxel features, a set of image features, and a set of point features and executing a panoptic decoder model using the set of voxel features, the set of image features, the set of point features, and a set of queries to generate a semantic mask and a track mask. The method further includes performing an action responsive to at least one of the semantic mask and the track mask.
G06V 10/80 - Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
G05D 1/242 - Means based on the reflection of waves generated by the vehicle
G05D 1/243 - Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals
G06V 10/26 - Segmentation of patterns in the image fieldCutting or merging of image elements to establish the pattern region, e.g. clustering-based techniquesDetection of occlusion
G06V 10/75 - Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video featuresCoarse-fine approaches, e.g. multi-scale approachesImage or video pattern matchingProximity measures in feature spaces using context analysisSelection of dictionaries
G06V 10/77 - Processing image or video features in feature spacesArrangements for image or video recognition or understanding using pattern recognition or machine learning using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]Blind source separation
G06V 10/82 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
13.
DEFERRED NEURAL LIGHTING IN AUGMENTED IMAGE GENERATION
Deferred neural lighting in augmented image generation includes performing operations. The operations include generating a source light representation of a real-world scene from a panoramic image of the real-world scene, augmenting the real-world scene in an object representation of the real-world scene to generate an augmented scene, and processing the augmented scene to generate augmented image buffers. The operations further include selecting a target lighting representation identifying a target light source, processing, by a neural deferred rendering model, the augmented image buffers, the source lighting representation, and a target lighting representation to generate an augmented image having a lighting appearance according to the target light source and outputting the augmented image.
Motion planning with implicit occupancy for autonomous systems includes obtaining a set of trajectories through a geographic region for an autonomous system, and generating, for each trajectory in the set of trajectories, a set of points of interest in the geographic region to obtains sets of points of interest. Motion planning further includes quantizing the sets of points of interest to obtain a set of query points in the geographic region and querying the implicit decoder model with the set of query points to obtain point attributes for the set of query points. Motion planning further includes processing, for each trajectory of a least a subset of trajectories, the point attributes corresponding to the set of points of interest to obtain a trajectory cost for the trajectory. From the set of trajectories, a selected trajectory is selected according to trajectory cost.
Imitation and reinforcement learning for multi-agent simulation includes performing operations. The operations include obtaining a first real-world scenario of agents moving according to first trajectories and simulating the first real-world scenario in a virtual world to generate first simulated states. The simulating includes processing, by an agent model, the first simulated states for the agents to obtain second trajectories. For each of at least a subset of the agents, a difference between a first corresponding trajectory of the agent and a second corresponding trajectory of the agent is calculated and determining an imitation loss is determined based on the difference. The operations further include evaluating the second trajectories according to a reward function to generate a reinforcement learning loss, calculating a total loss as a combination of the imitation loss and the reinforcement learning loss, and updating the agent model using the total loss.
Joint detection and prediction transformer includes functionality to perform operations that include obtaining a sensor data encoding of sensor data acquired from a sensor, initializing a spatiotemporal object structure, and obtaining a map encoding of a geographic region corresponding to the sensor data. The operations further include iteratively revising, by a transformer model, the spatiotemporal object structure using the sensor data encoding and the map encoding to generate a revised spatiotemporal object structure, decoding, by a decoder model, the spatiotemporal object structure to obtain location information and path information of at least one object in the geographic region, and outputting the location information and the path information.
A method includes generating a first sample including first raw parameter values of a first modifiable parameters by a probabilistic model and a kernel and executing a first test of a virtual driver of an autonomous system according to the first sample to generate a first evaluation result of multiple evaluation results. The method further includes updating the probabilistic model according to the first evaluation result and training the kernel using the first evaluation result. The method additionally includes generating a second sample including second raw parameter values of the parameters by the probabilistic model and the kernel and executing a second test of a virtual driver of an autonomous system according to the second sample to generate a second evaluation result of the evaluation results. The method further includes presenting the evaluation results.
G06F 30/27 - Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
G06F 11/36 - Prevention of errors by analysis, debugging or testing of software
LiDAR based memory segmentation includes obtaining a LiDAR point cloud that includes LiDAR points from a LiDAR sensor, voxelizing the LiDAR points to obtain LiDAR voxels, and encoding the LiDAR voxels to obtain encoded voxels. A LiDAR voxel memory is revised using the encoded voxels to obtain revised LiDAR voxel memory, decoding the revised LiDAR voxel memory to obtain decoded LiDAR voxel memory features. The LiDAR points are segmented using the decoded LiDAR voxel memory features to generate a segmented LiDAR point cloud.
Diffusion for realistic scene generation includes obtaining a current set of agent state vectors and a map data of a geographic region, and iteratively, through multiple diffusion timesteps, updating the current set of agent state vectors. Iteratively updating includes processing, by a noise prediction model, the current set of agent state vectors, a current diffusion timestep of the plurality of diffusion timesteps, and the map data to obtain a noise prediction value, generating a mean using the noise prediction value, generating a distribution function according to the mean, sampling a revised set of agent state vectors from the distribution function, and replacing the current set of agent state vectors with the revised set of agent state vectors. The current set of agent state vectors are outputted.
Real time image rendering for large scenes includes performing operations that include identifying a camera location of a camera in a geographic region, and rasterizing, using the camera location and a polygonal mesh, a UV feature map to obtain a feature buffer. The operations also include processing, by a shading machine learning model, the feature buffer using a view direction of the camera to generate an image rendering that includes opacity values and color values. The operations further include generating a rendered image from the opacity values and color values.
Compact LiDAR representation includes performing operations that include generating a three-dimensional (3D) LiDAR image from LiDAR input data, encoding, by an encoder model, the 3D LiDAR image to a continuous embedding in continuous space, and performing, using a code map, a vector quantization of the continuous embedding to generate a discrete embedding. The operations further include decoding, by the decoder model, the discrete embedding to generate modified LiDAR data, and outputting the modified LiDAR data.
Neural hash grid based sensor simulation includes interpolating hash grid features adjacent to a location in a neural hash grid defined for a target object to obtain a set of location features. A multilayer perceptron (MLP) model processes the set of location features to generate a set of image features for the location. The method further includes completing, using the set of image features, ray casting to the target object to generate a feature image, generating a rendered image from the feature image, and processing the rendered image.
A method includes obtaining, from sensor data, map data of a geographic region and multiple trajectories of multiple agents located in the geographic region. The agents and the map data have a corresponding physical location in the geographic region. The method further includes determining, for an agent, an agent route from a trajectory that corresponds to the agent, generating, by an encoder model, an interaction encoding that encodes the trajectories and the map data, and generating, from the interaction encoding, an agent attribute encoding of the agent and the agent route. The method further includes processing the agent attribute encoding to generate positional information for the agent, and updating the trajectory of the agent using the positional information to obtain an updated trajectory.
Unsupervised object detection from lidar point clouds includes forecasting a set of new positions of a set of objects in a geographic region based on a first set of object tracks to obtain a set of forecasted object positions, and obtaining a new LiDAR point cloud of the geographic region. A detector model processes the new LiDAR point cloud to obtain a new set of bounding boxes around the set of objects detected in the new LiDAR point cloud. Object detection further includes matching the new set of bounding boxes to the set of forecasted object positions to generate a set of matches, updating the first set of object tracks with the new set of bounding boxes according to the set of matches to obtain an updated set of object tracks, and filtering, after updating, the updated set of object tracks to remove object tracks failing to satisfy a track length threshold, to generate a training set of object tracks. The object detection further includes selecting at least a subset of the new set of bounding boxes that are in the training set of object tracks, and retraining the detector model using the at least the subset of the new set of bounding boxes.
Implicit occupancy for autonomous systems include receiving a request for a point attribute at a query point matching a geographic location, obtaining a query point feature vector from a feature map. The feature map encodes a geographic region that includes the geographic location. A first set of multilayer perceptrons of a decoder model process the query point feature vector to generate offsets. Offset feature vectors are obtained from the feature map for the offsets. A second set of multilayer perceptrons of the decoder model process the offset feature vectors and the query point feature vector to generate the point attribute. The operations further includes responding to the request with the point attribute.
Motion forecasting for autonomous systems includes obtaining map data of a geographic region and historical trajectories of agents located in the geographic region. The map data includes map elements. The agents and the map elements have a corresponding physical locations in the geographic region. Motion forecasting further includes building, from the historical trajectories and the map data, a heterogeneous graph for the agents and the map elements. The heterogeneous graph defines the corresponding physical locations of the agents and the map elements relative to each other of the agents and the map elements. Motion forecasting further includes modelling, by a graph neural network, agent actions of an agent of the agents using the heterogeneous graph to generate an agent goal location, and operating an autonomous system based on the agent goal location.
G06N 3/006 - Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
G06F 30/20 - Design optimisation, verification or simulation
27.
THREE DIMENSIONAL OBJECT RECONSTRUCTION FOR SENSOR SIMULATION
Three dimensional object reconstruction for sensor simulation includes performing operations that include rendering, by a differential rendering engine, an object image from a target object model, and computing, by a loss function of the differential rendering engine, a loss based on a comparison of the object image with an actual image and a comparison of the target object model with a corresponding lidar point cloud. The operations further include updating the target object model by the differential rendering engine according to the loss, and rendering, after updating the target object model, a target object in a virtual world using the target object model.
Real world object reconstruction and representation include performing operations that include sampling locations along a camera ray from a virtual camera to a target object to obtain a sample set of the locations along the camera ray. For each location of the at least a subset of the sample set, the operations include determining a position of the location with respect to the target object, executing, based on the position, a reflectance multilayer perceptron (MLP) model, to determine an albedo and material shininess for the location, and computing a radiance for the location and based on a viewing direction of the camera ray using the albedo and the material shininess. The operations further includes rendering a color value for the camera ray by compositing the radiance across the first sample set.
Trajectory value learning for autonomous systems includes generating an environment image from sensor input and processing the environment image through an image neural network to obtain a feature map. Trajectory value learning further includes sampling possible trajectories to obtain a candidate trajectory for an autonomous system, extracting, from the feature map, feature vectors corresponding to the candidate trajectory, combining the feature vectors into the input vector, and processing, by a score neural network model, the input vector to obtain a projected score for the candidate trajectory. Trajectory value learning further includes selecting, from the candidate trajectories, the candidate trajectory as a selected trajectory based on the projected score, and implementing the selected trajectory.
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
B60W 50/06 - Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
G05B 13/02 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
G06T 7/246 - Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
G06T 7/73 - Determining position or orientation of objects or cameras using feature-based methods
The operations of autonomous system training and testing may include generating a digital twin of a real world scenario as a simulated environment state of a simulated environment. The operations may also include iteratively, through multiple timesteps: executing a sensor simulation model on the simulated environment state to obtain simulated sensor output, obtaining, from a virtual driver of an autonomous system, at least one actuation action that is based on the simulated sensor output, updating an autonomous system state of the autonomous system based on the at least one actuation action, modeling, using multiple actor models, multiple actors in the simulated environment according to the simulated environment state to obtain multiple actor actions, and updating the simulated environment state according to the actor actions and the autonomous system state. The operations may furthermore include evaluating the virtual driver after updating the simulated environment state.
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
recorded computer software for operating self-driving and semi self-driving vehicles; downloadable computer software for the autonomous driving and semi-automated driving of motor vehicles providing online non-downloadable computer software for operating self-driving and semi self-driving vehicles; providing online non-downloadable computer software for the autonomous driving and semi-automated driving of motor vehicles; software development in the field of autonomous driving and semi-autonomous driving of motor vehicles
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
downloadable computer programs and software control programs for the assessment, simulation and development of autonomous vehicles providing temporary use of non-downloadable computer programs and software control programs for the assessment, simulation, and development of autonomous vehicles
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
(1) Recorded computer software for operating self-driving and semi self-driving vehicles; downloadable computer software for the autonomous driving and semi-automated driving of motor vehicles (1) Providing online non-downloadable computer software for operating self-driving and semi self-driving vehicles; providing online non-downloadable computer software for the autonomous driving and semi-automated driving of motor vehicles; software development in the field of autonomous driving and semi-autonomous driving of motor vehicles
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
recorded computer software for operating self-driving and semi self-driving vehicles; downloadable computer software for the autonomous driving and semi-automated driving of motor vehicles providing online non-downloadable computer software for operating self-driving and semi self-driving vehicles; providing online non-downloadable computer software for the autonomous driving and semi-automated driving of motor vehicles; software development in the field of autonomous driving and semi-autonomous driving of motor vehicles
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
(1) Computer software for operating self-driving and semi self-driving vehicles; Computer software for the autonomous driving and semi-automated driving of motor vehicles (1) Software development in the field of autonomous driving and semi-automated driving of motor vehicles
