Feature policies can control different aspects of features, such as creation of new objects, to provide fine-grained restrictions on how the features may be used. Feature policies are system policies that allow users to control creation of features within a given context, such as at the entire account level, specific database level, specific schema level, etc. Feature policies can be independent objects that can be bound and unbound to different hierarchical levels of an account. Multiple feature policies can be concurrently enforced.
Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for optimizing a query plan for executing a query by pushing down top-k information in the query plan with respect to a group-by grouping sets operator or a window function operator, which can be used within a data platform (e.g., database) environment. In particular, various example embodiments enable information from a top-k operator of a query plan to be pushed down the query plan and enable pushing the information to a select operator (e.g., aggregation operator) positioned below the top-k operator that implements the group-by grouping sets operator or the window function operator.
The described system aims to reduce or eliminate inaccuracies and hallucinations in responses generated by a machine learning model when processing user queries. The data platform parses and categorizes the text within data files to create structured textual representations. The user submits multiple prompts which are collectively assessed to refine and modify the initial queries. The modified query is used to identifying segments of data files that are most relevant to the query. These relevant portions are then compiled into a Retrieval-Augmented Generation (RAG) context block. This RAG context block is fed into a prompt response machine learning model, which processes the enriched information to generate a well-informed and accurate response to the user's query. Finally, this response is displayed back to the user through the chat interface, completing a cycle that enhances the reliability and relevance of machine-generated answers.
Optimal degree of parallelism (DOP) can improve query execution performance. An execution plan to execute the query may be generated, the execution plan comprising a plurality of operators and links. The execution plan may be decomposed into a plurality of sequences of connected operators of the plurality of operators. For each sequence, a set of input values may be determined, and a per-sequence optimal DOP value and a per-sequence minimum DOP value may be generated based on the set of input values for the respective sequence.
42 - Scientific, technological and industrial services, research and design
Goods & Services
Software as a service (SAAS) services featuring software for
database processing in the field of digitalization, software
development, business intelligence, data analytics,
artificial intelligence, and optimization; software as a
service (SAAS) featuring software for use in database
management; development and maintenance of computer software
platforms for artificial intelligence used to generate
business intelligence and recommendations; software as a
service (SAAS) services featuring software for data modeling
for use as business intelligence tool; data mining;
providing temporary use of non-downloadable software for use
in predictive analytics; providing temporary use of
non-downloadable software for use in data mining,
processing, analysis, visualization and information sharing;
software as a service (SAAS) services featuring software for
strategic planning, operational planning, providing business
intelligence, sharing business intelligence, analyzing
business intelligence, and natural language queries;
providing online non-downloadable virtual assistant software
that uses artificial intelligence for simulating
conversations; providing online non-downloadable virtual
assistant software that uses artificial intelligence for use
in creating applications for replying to questions from
businesses related to business data and analytics; platform
as a service (PaaS) services featuring software featuring a
virtual assistant that uses artificial intelligence and
natural language understanding to converse with users,
execute user instructions, execute queries, and index data;
software as a service (SaaS) services featuring software for
use in providing a virtual assistant that uses artificial
intelligence to enhance access to information and services
on a technology platform and website; none of the aforesaid
in connection with computer hardware incorporating
artificial intelligence (AI) technology, namely, integrated
circuits, microprocessors and microprocessor cores, and
computer software featuring artificial intelligence used in,
and for use in the design and development of, integrated
circuits, microprocessors, and microprocessor cores.
6.
BOTTOM-UP PLAN RECONSTRUCTION FOR SQL QUERY OPTIMIZATION
The subject technology receives an initial query plan. The subject technology receives a directive specifying a join order hint. The subject technology performs a bottom-up plan reconstruction process on the initial query plan based on the directive, the bottom-up plan reconstruction process generating a transformed query plan that conforms to the join order hint specified in the directive. The subject technology generates a transformed query plan that conforms to the join order hint specified in the directive. The subject technology provides the transformed query plan to an execution node for executing the transformed query plan.
This disclosure provides methods and techniques of using a data model to provide contact information for associating administration and management identities with data assets. An example method includes receiving a query request from a user for a first data object within a multi-tenant cloud-based data environment. The method further includes determining an absence of an access permission associated with the user based on security policies of the first data object. The method then identifies a second data object (e.g., a contact or contact object data model) associated with the first data object. The second data object includes communication information of an administrator (e.g., an owner, an assigned administrator, a system manager, or an inherent administrator, etc.) that has authority to grant the access permission to the user. The method further includes providing the communication information of the second data object to the user.
A cloud data platform determines a set of candidate pre-provisioning times for compute resources to be provisioned from a cloud provider service to a free pool maintained by the cloud platform, the set of candidate pre-provisioning times being based on a demand of compute resources (N) and a target time. For each candidate pre-provisioning time: the cloud data platform finds similar historical periods based on historical distribution of provisioning latencies. For each historical period of the similar historical periods, the cloud data platform determines a minimum number of compute resources (M) to provision at the candidate pre-provisioning time so that at least N compute resources are ready by the target time within a specified confidence.
09 - Scientific and electric apparatus and instruments
Goods & Services
Downloadable computer software for managing, deploying, and operating cloud data platform services, data engineering, data warehousing, data management, data mining, data monitoring, data optimization, data storage, data processing, data capture, data collection, secure data sharing, business intelligence, artificial intelligence, and data analytics; Downloadable database management software for loading data into and unloading data out of database tables, streaming and ingesting data into cloud-based data warehouses, querying databases, and accessing databases; Downloadable computer software for use in data migration; Downloadable computer software using artificial intelligence (AI) for use in connection with data processing, data storage, data capture, data collection, data warehousing, data management, data mining, database analytics, secure data sharing and cloud management; Downloadable computer software development tools and frameworks for building, developing, deploying and managing software applications in the fields of data science, artificial intelligence, machine learning and deep learning; Downloadable computer software for building, managing, and operating artificial intelligence models, machine learning models, and large language models; Downloadable computer software for integrating cloud data platforms with third-party data processing, development, and enterprise application platforms
09 - Scientific and electric apparatus and instruments
Goods & Services
Downloadable computer software for managing, deploying, and operating cloud data platform services, data engineering, data warehousing, data management, data mining, data monitoring, data optimization, data storage, data processing, data capture, data collection, secure data sharing, business intelligence, artificial intelligence, and data analytics; Downloadable database management software for loading data into and unloading data out of database tables, streaming and ingesting data into cloud-based data warehouses, querying databases, and accessing databases; Downloadable computer software for use in data migration; Downloadable computer software using artificial intelligence (AI) for use in connection with data processing, data storage, data capture, data collection, data warehousing, data management, data mining, database analytics, secure data sharing and cloud management; Downloadable computer software development tools and frameworks for building, developing, deploying and managing software applications in the fields of data science, artificial intelligence, machine learning and deep learning; Downloadable computer software for building, managing, and operating artificial intelligence models, machine learning models, and large language models; Downloadable computer software for integrating cloud data platforms with third-party data processing, development, and enterprise application platforms
Various example embodiments described herein provide for systems, methods, devices, instructions, and like for translation to semantic data for artificial intelligence (AI) agent-based semantic data updates (e.g., improvements) using one or more data sources, such as golden data sources, where the updated semantic data (e.g., semantic model) that results can be used by a natural language query system to translate one or more natural language questions into one or more structure language queries (e.g., one or more structured query language (SQL) statements). Some example embodiments are implemented as a system that uses a set of AI agents working in an optimization loop to improve/refine semantic data using the one or more data sources.
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
Downloadable software for data processing, data engineering, data modeling, data management, data mining, business intelligence, data analytics, predictive analytics, software development, and artificial intelligence; Downloadable computer software for building, managing, and operating artificial intelligence models, machine learning models, and large language models; Downloadable software for AI agents for autonomous and supervised task completion; Downloadable software using artificial intelligence for code generation, code completion, debugging, software development, AI agent management, and interaction with AI agents and AI-powered development tools; Downloadable software for AI-assisted and autonomous software development, including code generation, code editing, software development workflow automation, and task execution; Downloadable computer software for use in processing and generating natural language queries; Downloadable software for conversion of text and data files into software code, detecting, recognizing, processing, analyzing, understanding, generating and editing software code; Downloadable computer software using artificial intelligence (AI) for the production of speech, text, images, video, sound, and code; Downloadable chatbot software for simulating conversations, analyzing images, sound and video, summarizing text, creating content, generating code, brainstorming, and answering queries; Downloadable computer application software for mobile devices using artificial intelligence for simulating real time conversations, question-answering systems, searching and retrieving information, and providing reference materials and other resources available on computer networks; Downloadable software for use as an application programming interface (API) for accessing artificial intelligence software for code generation, software development, and AI agent interaction Software as a service (SAAS) services featuring software for data processing, data engineering, data modeling, data management, data mining, business intelligence, data analytics, predictive analytics, software development, and artificial intelligence; Software as a service (SAAS) services featuring software for building, managing, and controlling an AI agent and an extensive library of agent skills to manage projects and tasks; Software as a service (SAAS) services featuring software for processing, generating, understanding and analyzing natural language using artificial intelligence; Artificial intelligence as a service (AIAAS) services featuring software using artificial intelligence (AI) for building, deploying, optimizing, and monitoring autonomous AI agents; Platform as a service (PAAS) featuring computer software platforms for building, deploying, optimizing, and monitoring autonomous AI agents; Providing on-line non-downloadable software for analyzing, storing, managing, recording, authenticating, transmitting, and receiving information and data for use in autonomous AI agent platforms; Providing online non-downloadable software for conversion of text and data files into software code, and for detecting, recognizing, processing, analyzing, understanding, generating, and editing software code; Providing online non-downloadable virtual assistant software that uses artificial intelligence for simulating conversations; Platform as a Service (PaaS) services featuring software featuring a virtual assistant that uses artificial intelligence and natural language understanding to converse with users, execute user instructions, execute queries, and index data; Software as a service (SAAS) services featuring software for use in providing a virtual assistant that uses artificial intelligence to enhance access to information and services on a technology platform and website; Software as a service (SAAS) services featuring software for mobile devices using artificial intelligence for simulating real-time conversations, question-answering systems, searching and retrieving information, and software development assistance; Providing application programming interface (API) access to artificial intelligence software for code generation, software development assistance, and AI agent management; Software as a service (SAAS) services featuring software for use in computer programming and software development, namely, AI-powered coding agents for automated code generation, code completion, code editing, debugging, and code review; Providing online non-downloadable software featuring artificial intelligence and machine learning software for managing, providing and analyzing Large Language Model (LLM) framework for coding, including datasets and parameters; Software platforms for orchestrating and managing coding agents and autonomous software development agents within multi-agent workflows and agentic pipelines
Described is a system for optimizing LLM inference by modifying the LLM to skip processing of a subset of layers for input tokens. The data platform determines a modified architecture that identifies and bypasses specific layers and training the modified LLM through a distillation process to configure parameter weightings for the adjusted architecture. Upon receiving a text input query from a user, the data platform processes the query using the modified LLM, leveraging the layer-skipping mechanism to improve computational efficiency without compromising output quality. The processed query generates a response, which is subsequently displayed to the user.
G06F 16/387 - Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
Downloadable software for data processing, data engineering, data modeling, data management, data mining, business intelligence, data analytics, predictive analytics, software development, and artificial intelligence; Downloadable computer software for building, managing, and operating artificial intelligence models, machine learning models, and large language models; Downloadable software for AI agents for autonomous and supervised task completion; Downloadable software using artificial intelligence for code generation, code completion, debugging, software development, AI agent management, and interaction with AI agents and AI-powered development tools; Downloadable software for AI-assisted and autonomous software development, including code generation, code editing, software development workflow automation, and task execution; Downloadable computer software for use in processing and generating natural language queries; Downloadable software for conversion of text and data files into software code, detecting, recognizing, processing, analyzing, understanding, generating and editing software code; Downloadable computer software using artificial intelligence (AI) for the production of speech, text, images, video, sound, and code; Downloadable chatbot software for simulating conversations, analyzing images, sound and video, summarizing text, creating content, generating code, brainstorming, and answering queries; Downloadable computer application software for mobile devices using artificial intelligence for simulating real time conversations, question-answering systems, searching and retrieving information, and providing reference materials and other resources available on computer networks; Downloadable software for use as an application programming interface (API) for accessing artificial intelligence software for code generation, software development, and AI agent interaction Software as a service (SAAS) services featuring software for data processing, data engineering, data modeling, data management, data mining, business intelligence, data analytics, predictive analytics, software development, and artificial intelligence; Software as a service (SAAS) services featuring software for building, managing, and controlling an AI agent and an extensive library of agent skills to manage projects and tasks; Software as a service (SAAS) services featuring software for processing, generating, understanding and analyzing natural language using artificial intelligence; Artificial intelligence as a service (AIAAS) services featuring software using artificial intelligence (AI) for building, deploying, optimizing, and monitoring autonomous AI agents; Platform as a service (PAAS) featuring computer software platforms for building, deploying, optimizing, and monitoring autonomous AI agents; Providing on-line non-downloadable software for analyzing, storing, managing, recording, authenticating, transmitting, and receiving information and data for use in autonomous AI agent platforms; Providing online non-downloadable software for conversion of text and data files into software code, and for detecting, recognizing, processing, analyzing, understanding, generating, and editing software code; Providing online non-downloadable virtual assistant software that uses artificial intelligence for simulating conversations; Platform as a Service (PaaS) services featuring software featuring a virtual assistant that uses artificial intelligence and natural language understanding to converse with users, execute user instructions, execute queries, and index data; Software as a service (SAAS) services featuring software for use in providing a virtual assistant that uses artificial intelligence to enhance access to information and services on a technology platform and website; Software as a service (SAAS) services featuring software for mobile devices using artificial intelligence for simulating real-time conversations, question-answering systems, searching and retrieving information, and software development assistance; Providing application programming interface (API) access to artificial intelligence software for code generation, software development assistance, and AI agent management; Software as a service (SAAS) services featuring software for use in computer programming and software development, namely, AI-powered coding agents for automated code generation, code completion, code editing, debugging, and code review; Providing online non-downloadable software featuring artificial intelligence and machine learning software for managing, providing and analyzing Large Language Model (LLM) framework for coding, including datasets and parameters; Software platforms for orchestrating and managing coding agents and autonomous software development agents within multi-agent workflows and agentic pipelines
Application specification objects may be created by an application and are visible to consumer administrators with appropriate privileges. An application may be configured to create of one or more sensitive objects, which may expose the customer of the application to risk. Application specifications reduce friction in that the customer does not need to take the steps to create these sensitive objects. The application may create the objects directly, but the customer may instead just review and approve the use of those objects.
A data platform is provided that stores a hierarchical database including a database object and a set of child objects in a hierarchical parent-child structure, configures a respective replicable parameter of a set of replicable parameters for the database and each child object within the hierarchical database, and selectively replicates a set of objects of the database between accounts while maintaining data consistency during replication using the set of replicable parameters. The data platform implements parameter-based replication by introducing a customer-visible parameter that can be set at both database and child object levels, with child objects automatically inheriting replication settings from their parents unless explicitly overridden. The data platform maintains data consistency by checking domain support before replication, tracking inheritance relationships, and preserving non-replicated objects during failover operations unless specific conflicts exist.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
A replication engine is provided that receives a set of interface implementations from features in a distributed data platform, where each interface implementation provides replication constraints. The replication engine constructs a dependency graph based on relationships between the features using the replication constraints and determines a synchronization order for replication operations using the dependency graph. The replication engine executes the replication operations according to the determined synchronization order by coordinating between multiple handlers including a snapshot handler for primary-side operations, an association handler for managing cross-cutting relationships, and a synchronization handler for secondary-side operations.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
18.
Push down top-k information for query plan optimization
Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for optimizing a query plan to execute a query by pushing down top-k information in the query plan, which can be used within a data platform (e.g., database) environment. In particular, various example embodiments enable information from a top-k operator of a query plan to be pushed down the query plan and enable pushing the information through one or more outer join operators (e.g., through multiple levels of outer join operators) of the query plan to a select operator, such as an aggregation operator, positioned below the one or more outer join operators.
Techniques for facilitating data transfer from a deployment in an untrusted zone to a deployment in a trusted zone are described. In the cloud storage location in the trusted zone, the data may be arranged in an external table. The data can be scanned for malicious content by the data system in a more efficient manner because the data is arranged in the external table. After scanning is complete and no malicious content is detected, the data can be transferred to a secure account in the deployment in the trusted zone before it can be transmitted to other parts of the data system, providing another layer of security.
Provided herein are systems and methods for configuring disaster recovery for data sharing. A method includes configuring a first failover group (FG) with a first set of data objects as a primary FG at a first deployment of a network-based database system, and configuring a first replication group (RG) with a second set of data objects as a current primary RG within the first FG. The first FG is replicated from the first deployment to a second FG in a second deployment of the network-based database system. The second FG includes a second RG as a replica of the first RG. The second FG is configured as the primary FG and the second RG as the current primary RG in the second deployment based on detecting a failover event in the first deployment. The method includes performing data replication based on the second RG being the current primary RG.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
A system includes at least one hardware processor and at least one memory storing instructions that cause the at least one hardware processor to perform operations, such as configuring, at a first computing node, a feature registry based on a plurality of feature definitions received via a user interface associated with an application programming interface. The operations include obtaining, at a second computing node, decoded raw data comprising streaming data and batch data, and performing incremental computation of features using a dependency graph comprising dynamic table objects and using configuration data from the feature registry. The operations include maintaining, as a feature store, a low-latency serving storage for computed feature values and a versioned historical storage. The operations include updating the low-latency serving storage using an event-driven task, controlling retrieval by a computing node based on a retrieval application programming interface, and backfilling historic results using a version-aware query.
A system enables clients to utilize resources that have been committed from one or more third-party resource providers, allowing them to execute various operations on a data platform. The system tracks the aggregate historical use of these resources over a specific period and monitors any unused resources within a first term commitment over an initial time period. The system also tracks requests for additional resources representing times when demand exceeded the baseline commitment and required supplemental, short-term resources. This collected data is then fed into a predictive model. The model analyzes these inputs to generate a forecast of future resource needs, providing an informed view of anticipated demand for upcoming periods. Based on this forecast, the system executes a first term commitment for resources, securing the necessary capacity over a second time period in alignment with the forecasted usage, thus optimizing resource allocation and cost efficiency.
Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for performing commit operations (e.g., write operations) with recovery on a data system, such as a database system, that is interacting with a data source external to the data system, such as an external catalog. For some example embodiments, the database system generates an operation identifier in association with each command to commit a change to an entity of the catalog linked database, where the operation identifier serves as a unique tracking mechanism for commit operations across the distributed systems.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
The subject technology identifies a change interval for a table. The subject technology determines a set of unregistered partitions and a set of registered partitions within the change interval. The subject technology identifies one or more partition lineage graphs (PLGs) connecting partitions in the set of unregistered partitions with partitions in the set of registered partitions. The subject technology determines a first set of changes from one or more PLGs. The subject technology determines a second set of changes from a set of remaining partitions in the set of unregistered partitions with partitions in the set of registered partitions. The subject technology consolidates the first set of changes determined from one or more PLGs and the second set of changes from the set of remaining partitions to generate a final set of changes for the table within the change interval.
Disclosed are techniques for dynamically analyzing applications submitted for sharing on a data sharing platform using application security profiles. Application behavior information of the application may be obtained and a provider application security profile indicating a structure of and inputs to the application may be generated based thereon. If the application does not contain malicious code based on a scan of the provider application security profile, variations of the inputs to the application may be generated and the application may be run with the variations of the inputs to generate updated application behavior information. A replay application security profile may be generated based on the updated application behavior information. If the application does not contain any malicious code based on the scan of the replay application security profile, the application may be approved for listing on the data sharing platform.
G06F 21/57 - Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
G06F 21/56 - Computer malware detection or handling, e.g. anti-virus arrangements
Methods, systems, and computer programs are presented for installing and executing a user function in a cloud data platform. The system receives a function for execution, determines the dependent packages required, and caches these packages in the cloud data platform. Upon receiving a request to execute the function, the system prepares an execution environment by loading the cached dependent packages. The function is then executed utilizing the cached packages. The caching mechanism optimizes package installations and reduces latency, ensuring efficient and secure function execution. The system includes a sandbox environment for determining dependencies with restricted network access, enhancing security during the installation process.
G06F 21/53 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by executing in a restricted environment, e.g. sandbox or secure virtual machine
G06F 21/57 - Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
H04L 67/5683 - Storage of data provided by user terminals, i.e. reverse caching
Described is a system for enabling third-party listing providers to upload cloud service listings and updates to a data platform across remote servers. Each listing includes rich data such as metadata, allowing customers to access data offerings directly. When a listing provider submits an update to a listing, the system updates a central ground truth storage layer to maintain the latest version of each listing. An update notification is then sent to multiple remote servers. Upon receiving a pull request from a remote server, the system transmits the updated listing to that server, allowing it to display the latest version to customers.
G06F 9/50 - Allocation of resources, e.g. of the central processing unit [CPU]
G06F 21/62 - Protecting access to data via a platform, e.g. using keys or access control rules
H04L 67/1036 - Load balancing of requests to servers for services different from user content provisioning, e.g. load balancing across domain name servers
Techniques for resharing data in multi-tenant data system includes receiving, by a resharer account, a first listing for a first set of data shared by a provider account of the multi-tenant data system, the listing defining an access grant and a resharing grant for the first set of data. The resharer account creates a first database in the resharer account, the first database comprising virtual representations of the first set of data. The resharer account transforms the first set of data to generate a second set of data and stores the second set of data in a second database in the resharer account, the second set of data comprising at least one virtual representation of the first set of data. The resharer account publishes a second listing of the second set of data to share with a consumer account based on the resharing grant for the first set of data in the first listing.
Methods and systems for generating a global database profile are described. The methods and systems receive, from each of a plurality of layers of a database system, profiling data, the profiling data comprising a first set of profiling data received from a first layer of the plurality of layers and a second set of profiling data received from a second layer of the plurality of layers, the profiling data representing database logic operations and physical states at each of the plurality of layers. The methods and systems aggregate the profiling data received from the plurality of layers into global database profiling data associated with the plurality of layers of the database system and present visual output based on the profiling data, the visual output representing the database logic operations and the physical states at each of the plurality of layers over time.
Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for using AI model cascades to filter data on a data system, such as a database system, based on an artificial intelligence model prompt (e.g., user-provided prompt). In particular, various example embodiments enable a database system to use cascaded AI models and adaptive bounds to optimize data filtering operations based on an artificial intelligence model prompt (also referred to herein as just a prompt) while balancing computational cost and accuracy, which can be useful in processing large-scale data queries.
An input file is split into a multiple chunks. Each chunk is assigned to one of multiple worker nodes assigned to processing the input file. Refined boundaries for the multiple chunks are calculated. The refined boundaries define multiple adjusted chunks. The calculating of the refined boundaries comprises determining summary statistics for the multiple chunks, determining start positions of the multiple chunks based on the summary statistics, and determining end positions of the multiple chunks based on the start positions of the multiple chunks. Data scan passes are performed on the multiple adjusted chunks based on the refined boundaries to populate one or more data structures for storing information extracted from the input file.
42 - Scientific, technological and industrial services, research and design
Goods & Services
(1) Software as a service (SAAS) services featuring software for database processing in the field of digitalization, software development, business intelligence, data analytics, artificial intelligence, and optimization; software as a service (SAAS) featuring software for use in database management; development and maintenance of computer software platforms for artificial intelligence used to generate business intelligence and recommendations; software as a service (SAAS) services featuring software for data modeling for use as business intelligence tool; data mining; providing temporary use of non-downloadable software for use in predictive analytics; providing temporary use of non-downloadable software for use in data mining, processing, analysis, visualization and information sharing; software as a service (SAAS) services featuring software for strategic planning, operational planning, providing business intelligence, sharing business intelligence, analyzing business intelligence, and natural language queries; providing online non-downloadable virtual assistant software that uses artificial intelligence for simulating conversations; providing online non-downloadable virtual assistant software that uses artificial intelligence for use in creating applications for replying to questions from businesses related to business data and analytics; platform as a service (PaaS) services featuring software featuring a virtual assistant that uses artificial intelligence and natural language understanding to converse with users, execute user instructions, execute queries, and index data; software as a service (SaaS) services featuring software for use in providing a virtual assistant that uses artificial intelligence to enhance access to information and services on a technology platform and website; none of the aforesaid in connection with computer hardware incorporating artificial intelligence (AI) technology, namely, integrated circuits, microprocessors and microprocessor cores, and computer software featuring artificial intelligence used in, and for use in the design and development of, integrated circuits, microprocessors, and microprocessor cores.
A data platform is provided that builds a container image filesystem from a container image. The data platform executes an application that accesses the container image filesystem and the data platform records directory and file access activity of the container image filesystem by the application during execution. The data platform dynamically rebuilds the container image filesystem using the directory and file access activity, and mounts the container image filesystem as rebuilt during a subsequent execution of the application.
Described is a system for updating an LLM-generated SQL query preventing double counting by receiving a natural language query from a user at a cloud-based server, where the query requests access to data in a database. The data platform identifies semantic data from the query and, using a large language model (LLM), generates a SQL query containing join functions. The SQL query is then parsed to identify operation types and data sources. An undirected graph is constructed from the parsed query, with nodes representing data sources and edges representing join functions. The undirected graph is converted to a directed graph by adding relationship characteristics to the edges. Based on this directed graph, the data platform validates the SQL query to detect and resolve potential double counting issues and updates the SQL query as necessary to ensure accurate results.
Various example embodiments provide for threat modeling using machine learning models and context information, where a threat model is generated based on a threat model diagram for a target system being analyzed for threat risks/scenarios. For an individual threat model generated, a threat scenario (e.g., each individual threat scenario) described in the individual threat model can be processed (e.g., individually processed) by a plurality of machine learning models to determine a set of generic mitigation labels for the threat scenario, where each generic mitigation label corresponds to a generic mitigation strategy for mitigating the threat scenario. The set of generic mitigation labels for the threat scenario with context information can be processed by one or more large language models to generate a set of specific mitigation labels for the individual threat model, where each specific mitigation label corresponds to a specific mitigation strategy.
The subject technology stores an original file master key identifier (FMKID) in metadata associated with an encrypted file, the encrypted file including data. The subject technology determines that the encrypted file is to be re-encrypted based at least on a period of time since the original FMKID was stored. The subject technology re-encrypts the data using a new encryption key associated with a new FMKID. The subject technology generates a mapping from the original FMKID to the new FMKID. The subject technology stores the mapping in a metadata database, the mapping being associated with a re-encrypted file, the re-encrypted file including the re-encrypted data.
The subject technology provides embodiments for integrating data lake functionality within a relational database management system (RDBMS) through a specialized RDBMS Data Lake Module. The subject technology enables database systems to natively support data lake table formats using SQL syntax. The system implements dual-mode access patterns supporting both internal operations through standard RDBMS interfaces and external operations via a catalog API, enabling multiple systems to access the same data lake tables without duplication. The RDBMS Data Lake Module comprises three integrated components: a lake format engine for translating SQL queries to object storage operations, a catalog manager for maintaining transactional consistency, and a catalog API for external system coordination.
A data platform is provided that collects memory usage data for a function during execution of the function and determines memory constraints of the function during execution using a configurable statistical method and the memory usage data. The data platform dynamically adjusts a memory limit for the function based on the memory constraints and schedules a job including the function using the memory limit without user intervention.
Monitoring health metrics of computing devices in a data system can be implemented at different levels. At a first level, local background services can be run locally on the computing devices to monitor a set of health metrics on the respective computing devices. At a second level, a central health marker service can monitor a set of health metrics for the computing devices.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Various example embodiments described herein provide for systems, methods, devices, instructions, and like for translation to semantic data for a schema of a data store with selective metadata merging, where the semantic data can enable generation of a structured language data query for the data store based on a natural language question. In particular, some example embodiments access existing non-compatible data that describes a data store schema of a data store, access existing compatible semantic data or generate (new) compatible semantic data that describes the data store schema, and compare metadata between one or more structural elements of the existing non-compatible data and one or more corresponding structural elements of the (accessed/generated) compatible semantic data to selectively merge metadata into the compatible semantic data.
The subject technology receives a query, the query including a statement to perform a loading operation into a table. The subject technology compiles the query to generate an initial query plan. The subject technology determines a cardinality estimate of the loading operation into the table. The subject technology determines whether the cardinality estimate is greater than a first threshold value. The subject technology, in response to the cardinality estimate being greater than the first threshold value, updates the initial query plan to generate a query plan, the query plan including a native insert subplan and a bulk load subplan. The subject technology sends the query plan to an execution node for execution.
The subject technology receives a set of mapped party IDs. The subject technology performs a join operation on the set of mapped party IDs. The subject technology generates a first aggregated list of customer ID and related party CRM ID. The subject technology filters the first aggregated list. The subject technology receives a second aggregated list of customer ID and related party CRM ID. The subject technology determines a metric indicating a sharing propensity of a related party. The subject technology performs union and deduplicate operations on the first and second aggregated lists, and a set of industry influencers. The subject technology performs a lookup operation to determine whether a particular related party shares with a customer ID. The subject technology sorts a third list of recommendations based at least in part on each score of each related party. The subject technology provides for display a sorted list of recommendations.
A method for processing analytical-style queries on a transactional database is provided. A key-value database configured for point lookups and singular transactions is maintained. An analytical-style query request against the key-value database is received and analyzed to determine if it requires accessing more than a threshold amount of data. In response to a threshold, key-value data is separated into range granules covering specified data ranges. For each range granule, a snapshot file comprising key-value pairs at a specific database version and delta files specifying changes during time ranges are generated. The delta files are streamed via change feeds and buffered until accumulating a predetermined amount of mutations. The snapshot and delta files are provided to client execution nodes for parallel processing of the analytical-style query request.
G06F 16/2458 - Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
A system and method for optimizing data ingestion through clustering during the copy process. The system detects a data processing pipeline definition with a COPY command and determines storage location and destination table parameters. Upon detecting a COPY command, the system parses the command syntax to identify storage location and destination table parameters. The system monitors for available file batches at the storage location and executes the COPY command to transform these batches into formatted files compatible with the destination table format. The system then performs clustering operations on the formatted files, organizing data according to pre-defined clustering keys, before ingesting the clustered files into the destination table.
A data platform is provided that implements a private endpoint pinning. The method comprises providing a private endpoint service for a set of customer accounts within a database deployment. Upon receiving a request to register a private endpoint of the private endpoint service with a customer account, the data platform verifies ownership and access privileges of the private endpoint. The private endpoint is then pinned to the customer account by registering the pinning in a private account mapping data persistence object. When a data access request is received from the private endpoint for access to the customer account, the data platform verifies the pinning between the private endpoint and the customer account using the registration. Based on this verification, the data access request is allowed.
The subject technology dynamically decides between two approaches to produce change data capture (CDC) records depending on estimated cost. One approach corresponds to a time-travel based approach. In the other approach, the subject technology builds off the mutations recorded by the underlying linearizable store between the two timestamps to produce CDC.
09 - Scientific and electric apparatus and instruments
42 - Scientific, technological and industrial services, research and design
Goods & Services
Downloadable computer software for use in data migration; Downloadable computer software to facilitate the customization of cloud data platform, database, data warehousing, data management, data mining, data monitoring, data optimization and data analytics; downloadable computer software that manages network traffic into and out of databases Data migration services; Cloud computing featuring software for use in data migration, data backup, and data retrieval; Software as a service (SAAS) services featuring software for use in data migration, data backup, data warehousing, data management, data mining, data optimization, database analytics and data retrieval; Providing temporary use of non-downloadable software for data extraction, data integration, data management, data consolidation, data migration, data configuration, data unification and data loading; Platform as a service (PAAS) featuring computer software platforms for use by others in connection with data migration; Artificial intelligence as a service (AIAAS) featuring software using artificial intelligence for database management; Computer technology consultancy in the field of data integration, data transformation, data management, data migration, data warehousing, data sharing, data analytics, data mining and data security; Data warehousing; Consulting services in the field of cloud computing, namely, customized assessment of infrastructure migration, modernization and digital transformation for cloud computer systems
48.
ADAPTIVE MACHINE-LEARNING-MODEL PROCESSING FOR WORKLOAD TYPES
Described is a system for responding to an interactive request by receiving an interactive request from a user through a communication interface on a data platform, the request including a question needing a response. A priority query request is generated based on the user's input, with a defined latency requirement specifying the maximum time allowed for generating the response. In order to meet this latency requirement, the batch size for processing on a graphical processing unit (GPU) handling multiple workloads of a large language model (LLM) is reduced. The priority query request is then inserted into the current batch being processed by the GPU, which is adjusted based on the reduced batch size. The response is generated by the GPU and subsequently displayed to the user through the communication interface
Provided herein are systems, methods, and computer-storage media for credential-less access to remote resources. An example method includes creating, by at least one hardware processor, a security integration object. The security integration object includes an identification of a role associated with access to an external resource. The method includes creating a secret object comprising a token associated with the security integration object. The method includes binding the security integration object and the secret object to generate a storage integration object. The storage integration object includes an external access integration. The method includes retrieving a cloud provider token during execution of a user-defined function (UDF). The method includes granting the UDF access to the external resource based on authenticating the cloud provider token using the external access integration.
The subject technology receives a first query. The subject technology determines a set of rows that are modified based on performing the first DML operation on the first table, the set of rows including a first set of updated rows. The subject technology generates a first delta file based on the determined set of rows. The subject technology generates a first updated rows file based on a first set of updated rows. The subject technology determines a second set of rows of the first table that have not been modified by performing the first DML operation on the first table. The subject technology generates a first residual file based on the second set of rows. The subject technology performs a validation process to validate the first DML operation using at least the first delta file, the first residual file, and a matching CoW file.
The subject technology receives a first query, the first query comprising a first set of statements, the first set of statements including at least a first statement for performing a first Data Manipulation Language (DML) operation on a first table, the first table included in a source file.
The subject technology receives a first query, the first query comprising a first set of statements, the first set of statements including at least a first statement for performing a first Data Manipulation Language (DML) operation on a first table, the first table included in a source file.
The subject technology executes the first query, the executing including determining whether to perform a copy-on-write (CoW) process. subject technology performs a CoW fallback process. The subject technology performs the copy-on-write process for the first query based on a result of the CoW fallback process.
A system and method for enhancing data storage security in cloud environments using multiple gateway endpoints. The system includes a cloud data platform with a plurality of networks, each associated with a different gateway endpoint. Each gateway endpoint has a respective gateway endpoint tag. The method involves creating networks and corresponding gateway endpoints, configuring each gateway endpoint to enable traffic for sessions associated with the gateway endpoint tag, and assigning execution nodes to networks with session tokens that include the gateway endpoint tag. This setup ensures that only sessions with the correct tag can access data storage, reducing the risk of unauthorized data exfiltration.
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
H04L 67/63 - Routing a service request depending on the request content or context
Aspects of the present disclosure relate to configuring managed policy for internal stage network policy. More specifically, a method of the present disclosure includes generating and activating a set of network policies on a control plane of a service provider. The method includes determining to convert the set of network policies activated on the control plane of the service provider into an IAM policy at the service provider. The method includes utilizing at least one of the converted IAM policies of an active network policy in response to a creation of a presigned URL. The method includes determining whether an STS token is approved based on the presigned URL and at least one of the set of network policies or the IAM policy. The method includes utilizing an approved STS token to access the service provider.
Provided herein are systems, methods, and computer-storage media for managing data object access in a catalog service account. The method includes detecting, at a catalog service account, by a catalog manager (CM), a user request to access a data object. The user request includes identification information of the user and a principal role associated with the user. The CM retrieves one or more catalog roles from a first catalog of the catalog service account. The one or more catalog roles correspond to the principal role. The CM determines a set of hierarchical data objects based on the user request. The set of hierarchical data objects comprise the data object. The CM performs a determination of whether access privileges of the set of hierarchical data objects and access privileges of the one or more catalog roles allow access to the data object. The CM grants access to the data object based on the determination.
A system is disclosed comprising a memory containing instructions and one or more computer processors. When the instructions are executed, the system performs an operation to configure a Domain Name System (DNS) proxy, executing in a node of a cloud data platform associated with a first account, to perform hostname resolution of an Account Host Identifier (AHID) of the first account. The DNS proxy receives a DNS request from a process executing in a pod of the node, and the system fails to resolve the DNS request if the name in the DNS request differs from the AHID of the first account. The system returns an Internet Protocol (IP) address if the name in the DNS request matches the AHID. The process executing in the pod of the node is configured to send data to data storage of the cloud data platform using the returned IP address.
G06F 15/16 - Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
H04L 61/4511 - Network directoriesName-to-address mapping using standardised directoriesNetwork directoriesName-to-address mapping using standardised directory access protocols using domain name system [DNS]
56.
AI DATA CONNECTIVITY FOR UNSTRUCTURED DATA REPOSITORIES
Described is a system that receives data from a variety of external data repositories and identifies unstructured data within the received content. The unstructured data is processed to generate textual representations. A chat message is displayed in a user interface, prompting the first user to submit a query. Upon receiving the user's query, the system generates a modified version of the query and identifies portions of the textual representations. A content block is then generated from these portions and input into a machine learning model trained to generate responses using content blocks. The system generates a response to the user's query and displays the response within the user interface.
H04L 51/02 - User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
A top K query directed at a table is received. Run-time pruning is performed during execution of the top K query on the table. The run-time pruning comprises determining, by a top K node, a current boundary based on a set of values identified by a table scan node in scanning the table and applying, by the table scan node, the current boundary to prune data during the scanning of the table. The applying of the current boundary comprises reducing scanning ranges of the table scan node based on the top K column being a key column of the table and filtering values scanned by the table scan node based on the top K column being a non-key column of the table. The result set is returned responsive to the top K query based on the run-time pruning performed during execution of the top K query on the table.
Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for suffix-based speculative token decoding for an artificial intelligence model, such as a language model (e.g., large language model). In particular, some example embodiments provide an AI model system with hybrid speculative token decoding, which combines suffix-based speculative token decoding with a draft AI model approach to speculative token decoding. With this hybrid decoding approach, various example embodiments can accelerate inference throughput while adapting to different types of workloads, particularly agentic applications that exhibit repetitive token generation patterns.
A client system of a computing environment including a data platform is provided that optimizes a database query. The client system creates a logical plan tree for a Structured Query Language (SQL) query, with the logical plan tree comprising a set of nodes. The client system identifies a set of duplicate nodes in the set of nodes of the logical plan tree and identifies a duplicate subtree in the logical plan tree by determining a set of root nodes of the duplicate subtree using parent-child relationships of the set of nodes. The client system generates an optimized query by replacing instances of subqueries represented by the duplicate subtree using a set of optimized subqueries. The client system communicates the optimized query to the data platform for execution.
Embodiments of the present disclosure provide techniques for classification with automated model selection, tuning, and training. A processing device receives, from a client, a data query referencing an input data set of a database associated with a virtual warehouse. The processing device allocates an amount of memory of the virtual warehouse to be used to train a machine learning (ML) model based on the input data set and a peak memory estimate, where the peak memory estimate is based on a heuristic. The processing device trains, based on the input data set and the data query, the ML model in the virtual warehouse using the amount of memory.
Example methods include providing a data platform for participants to assign roles of various rights to access data objects on the data platform. A first participant acting in a role as an owner may create a code entity on the data platform to interact with the data objects. A second participant acting in a role as an administrator may define a security boundary over the code entity created by the first participant. A third participant acting as a caller may request to interact with the data objects using the code entity. A processing device of the database system provides the third participant, access to the code entity created by the first participant based on the security boundary defined by the second participant.
Methods, systems, and computer programs are presented for generating and managing query plan representations using a unified framework based on a common query plan configuration. The system includes a memory with instructions and one or more computer processors. The instructions, when executed, cause the system to provide a user interface (UI) for a development tool to create a configuration for a plan builder. The UI offers options to create a plurality of query plans, including a logical query plan, a physical query plan, a query plan hash, and a query plan signature. The system receives parameters to create the configuration and select a query plan from the plurality of query plans. The system generates a first query plan based on the parameters and causes the presentation of the first query plan on a display.
Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for optimizing a query plan to execute a query by using early row flush and pushing down top-k information in a query plan that includes an inner join, which can be used within a data platform environment. In particular, various example embodiments use early flush operations by one or more aggregation operators to eventually enable information from a top-k operator of a query plan to be pushed down the query plan and through one or more inner join operators of the query plan to one or more select operators (e.g., aggregation operators and table scan operators) positioned below the one or more inner join operators.
The described system aims to reduce or eliminate inaccuracies and hallucinations in responses generated by a machine learning model when processing user queries. The data platform parses and categorizes the text within data files to create structured textual representations. The user submits multiple prompts which are collectively assessed to refine and modify the initial queries. The modified query is used to identifying segments of data files that are most relevant to the query. These relevant portions are then compiled into a Retrieval-Augmented Generation (RAG) context block. This RAG context block is fed into a prompt response machine learning model, which processes the enriched information to generate a well-informed and accurate response to the user's query. Finally, this response is displayed back to the user through the chat interface, completing a cycle that enhances the reliability and relevance of machine-generated answers.
Techniques for managing states of virtual warehouses in a multi-tenant network-based data system are described. A “resolver” may be provided in each warehouse scheduling service thread. The resolver may maintain a current state of the virtual warehouse and may generate a target state of the virtual warehouse based on an operation request, such as a resume operation, a suspend operation, resize operation, etc. The resolver may generate an action plan to converge the current state to the target state.
Questionnaire completion systems and methodologies for a data platform. The data platform receives from a consumer an unstructured questionnaire to be completed based on structured database objects, semi-structured database objects, and unstructured database objects stored on the data platform by a provider. The data platform generates a secured completion of the unstructured questionnaire based on a questionnaire completion model and the unstructured questionnaire. The data platform determines a confidence score for the completion and in response to determining the confidence score does not exceed a threshold value, the data platform generates a structured query based on the unstructured questionnaire and a structured query model, and generates the secured completion based on querying the structured database objects using the structured query. The data platform applies a security function to the secured completion to generate a completion of the unstructured questionnaire and provides the completion to the consumer.
A query engine can use partition-granular level statistics to optimize query performance. A query can reference a table with a plurality of partitions and include a predicate. A partition-granular selectivity estimate for the predicate can be generated based on statistics stored regarding the plurality of partitions of the table. A query plan can be generated based on partition-granular selectivity estimate to optimize query processing.
Methods, systems, and computer programs are presented for providing performance metrics in an online performance analysis system employing customer objects, such as database tables. A plurality of metric source data associated with a plurality of objects is accessed and a subset of the plurality of objects is determined that satisfies stableness criteria based on the plurality of metric source data to identify a set of stable objects. A set of metrics is generated based on the subset of the plurality of objects that satisfies the stableness criteria.
Antagonistic queries can have a high resource and time footprint triggering a range of issues, such as causing compilation performance degradation of other queries and machine failures. Described herein are techniques for automatically identifying antagonistic queries and redirecting the antagonistic queries to dedicated resources. This redirecting can help better balance the workload on different work clusters and to isolate antagonistic workloads from impacting the compilation and execution performance of other queries.
Various embodiments described herein provide for systems, methods, devices, instructions, and like for generating synthetic data. According to various embodiments, synthetic data generation comprises receiving input specifying one or more source tables and join key columns, and generating synthetic data that preserves statistical similarity and referential integrity among columns of the source data.
A system is disclosed for recovering historical table data in a database environment. The system includes at least one hardware processor and at least one memory. The memory stores instructions that, when executed, cause the system to receive a request to recover historical table data of a source table. The historical table data includes multiple partition files, and each partition file includes a deleted file designation. The system performs a recovery process on the partition files by determining a recoverable time range for the source table based on lifecycle information and restoring the partition files based on the recoverable time range. The system retrieves a schema associated with the historical table data and generates metadata corresponding to the schema. The metadata is associated with the recovered partition files to reconstruct the historical table data. This approach allows efficient and reliable recovery of deleted or lost table data.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
A data platform including an error handling framework for loading of input data. The data platform generates input data columns based on an input file and generates result data columns based on the input data columns and evaluating expressions. The data platform detects projection errors during the generating of the result data columns and stores result error indicators in error indicator arrays of the result data columns based on the projection errors. The data platform generates filtered result data columns based on the result data columns and the result error indicator arrays of the result data columns and stores the filtered result data columns in a database of the data platform.
Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for structured language parsing to execute a differentially private query on a database system. According to some example embodiments, a user (e.g., an analyst) submits to a data system (e.g., data platform) a differentially private query using a structured language interface (e.g., SQL interface), which causes the calling of one or more stored procedures on the data system, where the one or more stored procedures encapsulate or facilitate use of a differential privacy engine, which can execute the differentially private query and generate a differentially private query result.
A system is disclosed that includes one or more hardware processors and at least one memory storing instructions. The system receives a first query directed towards a shared dataset and accesses a first set of data from a first table in the shared dataset. The system determines that an aggregation constraint policy is attached to the first table, which restricts output of data values stored in the table. The system performs a uniqueness check on join keys for a join operation associated with the first table, verifying that at least one row from the first table is not amplified in the result. The system enforces the aggregation constraint policy on the first query based on this verification. The system generates an output to the first query based on the first set of data. This approach helps control data aggregation and ensures privacy when accessing shared datasets.
Systems and methods are provided for creating a secure database execution environment. The system generates, by a database system executing on a secure enclave, attestation information. The system transmits the attestation information to a remote entity. The system obtains, by the database system executing on the secure enclave, one or more encryption keys in response to the remote entity authenticating the attestation information. The system performs, by the database system executing on the secure enclave, one or more database operations on encrypted data stored on the database system using the one or more encryption keys.
Described is a system for join constraints for query processing by receiving a first query directed towards a shared dataset in a data clean room; assessing the first query to identify that the one or more functions s at least a join function; determining that the first query is configured to join a first set of data from the shared dataset with a second set of data using the join function; determining that a join constraint policy is to be enforced in relation to the first query; and generating an output to the first query based on the execution of the one or more functions, the output to the first query without data values stored in the portion of the first set of data based on determining that the join constraint policy is to be enforced in relation to the first query.
The subject technology sends, from a child rowset operator (RSO) instance, a first request for performing a user defined aggregate function (UDAF) to a user defined function (UDF) server to initialize an aggregate state for a set of aggregation groups and update aggregated states for each aggregation group from the set of aggregation groups, the first request including a set of input rows. The subject technology receives information comprising a computation status of the UDAF. The subject technology sends, by the child RSO instance, a second request to the UDF server to update the aggregated states for each aggregation group from the set of aggregation groups, the second request including a second set of input rows. The subject technology receives an aggregate states vector with one entry per aggregation group. The subject technology sends, by the child RSO instance, the aggregate states vector to a parent RSO instance.
A join decision manager (JDM) generates a data processing pipeline. The data processing pipeline includes at least one join operation associated with build-side row data and probe-side row data. The JDM determines the maximum cardinality associated with the probe-side row data. The JDM determines size of the build-side row data at a decision node of the at least one join operation. The JDM configures execution of the at least one join operation as one of a broadcast join or a hash-hash join based on the size of the build-side row data and the maximum cardinality.
Disclosed are techniques for routing and filtering telemetry data based on custom telemetry definitions provided by a user. A telemetry filter definition comprising rules for routing and filtering telemetry data may be converted into a common expression language (CEL) abstract syntax tree (AST). The CEL AST may be provided to a filtering component, which may compile the CEL AST into a CEL filter program comprising the rules for routing and filtering telemetry data. In response to receiving telemetry data, filtering, by the filtering component, the received telemetry data based on the CEL filter program to generate filtered telemetry data.
Systems and methods are provided for controlling the deletion of data in a database system. The system receives input comprising a deletion criterion for a database system. The system applies the deletion criterion to a set of tables of the database system. The system determines that an individual portion of the set of tables satisfies the deletion criterion. In response to determining that the individual portion of the set of tables satisfies the deletion criterion, the system transfers the individual portion of the set of tables to a temporary storage system.
Disclosed is an execution information sharing system that writes execution information to a provider target (and other targets) in a secure manner. Execution information generated by an application may be written to a consumer stage, wherein the application is shared by a provider account of a data exchange with a consumer account that executes the application. A consumer exchange service(ES) of the data exchange may send a request to a copy service of the data exchange to copy the execution information from the consumer stage to the provider stage, wherein the consumer ES is a part of the data exchange and is protected from actions of the consumer account. A copy operation may be executed to copy the execution information from the consumer stage to the provider stage using the copy service of the data exchange. The execution information is ingested from the provider stage to a provider table.
Disclosed are techniques for using an application control framework to build, share and manage access to and usage of applications via a data sharing platform. An application control framework may provide a number of predefined controls and may receive values for certain predefined controls as well as custom control definitions and corresponding values from a provider. The application control framework may also receive application logic and may build an application package comprising the application logic and a set of controls including the predefined and custom controls to manage access to and usage of the application. In response to a consumer of the data sharing platform importing the application package, the application control framework may call the set of install scripts to install an instance of the application in the consumer account using the application logic and manage access to the application instance by the consumer using the set of controls.
A hot server is identified from a plurality of servers based on one or more server metrics associated with the hot server. A hot data range stored by the hot server is identified based on one or more read density metrics. The hot data range comprises a range of data values with a higher volume of access requests compared to other data values stored by the hot server. The hot data range is replicated across a number of additional servers.
G06F 16/00 - Information retrievalDatabase structures thereforFile system structures therefor
G06F 9/50 - Allocation of resources, e.g. of the central processing unit [CPU]
G06F 16/25 - Integrating or interfacing systems involving database management systems
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
Embodiments of the present disclosure provide techniques for efficient computation over a wide table. A processing device determines that a first number of columns of a first table is greater than a threshold number of columns. The processing device transforms the first table into a second table based on the determination, where the second table includes a second number of columns that is less than the first number of columns, and where the second table includes a first column that includes first fields that identify columns of the first table, a second column that includes second fields that identify data types of fields of the first table, and a third column that includes third fields that include data of the fields of the first table. The processing device executes a UDTF on the second table.
Cloning operations can be used generate snapshots of tables at specified times. The snapshot objects can be stored in a first-tier storage with the table, where the cloned versions of the tables and the table may share files, such as micro-partition files, to conserve storage resources. After a first expiration time, snapshot objects can be transferred from the first-tier storage to a second-tier storage to further save on storage costs. After a second expiration time (e.g., full retention period), the snapshot objects can be deleted from the second-tier storage as well.
A data platform is provided that uses a replication cache to replicate data. The data platform is designed to receive a replication request from a secondary deployment that includes a request for a data transfer of data files from a primary deployment. The data platform analyzes metadata of a replication cache and the primary deployment to identify the data files for replication. Based on this metadata, the data platform determines whether to route the data transfer through the replication cache or directly from the primary deployment to the secondary deployment. The data transfer is then routed accordingly, and the receipt of the data transfer at the secondary deployment is verified.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
G06F 11/34 - Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation
Systems and methods for replicating unstructured staged data between remote database deployments are disclosed. The system includes at least one hardware processor and memory storing instructions that identify staged data at a first database deployment for replication to a second, remote database deployment. The staged data includes unstructured data items stored in a storage resource associated with the first database deployment. The system replicates a directory from the first database deployment to the second, where the directory includes information identifying the unstructured data items. Metadata is also replicated, including references to the locations of the unstructured data items in the storage resource. The second database deployment is enabled to access one or more unstructured data items from the storage resource of the first database deployment using the directory and references, without duplicating the data. Incremental replication of additional staged data is facilitated based on a comparison of directories between deployments.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
G06F 16/25 - Integrating or interfacing systems involving database management systems
88.
DYNAMIC TABLES WITH EXTERNALLY MANAGED ICEBERG SOURCE TABLES
Provided herein are systems and methods for configuring dynamic tables with externally-managed Iceberg source tables. An example method for updating a dynamic table using data from an Iceberg source table includes generating, for each row in an Iceberg source table, a row identifier derived from immutable metadata associated with a physical storage location of the row and a position of the row within the physical storage location. The method further includes generating, for each of a first version and a second version of the Iceberg source table, a set of the row identifiers by computing the row identifier for each row present in the respective version. The sets of the row identifiers are compared between the first version and the second version of the Iceberg source table to identify changes at a row level. A dynamic table associated with the Iceberg source table is updated based on the identified changes.
To provide outbound private link support for a multi-tenant data system with tenant isolation, a separate, dedicated virtual network is provided, referred to as private link (PL) virtual network. The PL virtual network may host a plurality of host interface endpoints and resource endpoints. A core virtual network and the PL virtual network may be peered together to work in conjunction. The private endpoints in the PL virtual network may then be connected to external systems using a private link without exposure to the public internet.
A system includes at least one hardware processor and memory storing instructions. The processor generates a query plan for a received query. The query plan includes multiple hash-join-build and hash-join-probe operations. A primary decision node is configured in the query plan. The primary decision node receives build-side data information from the hash-join-build operations. For each hash-join-build operation, a memory amount for performing a broadcast is determined. A subset of hash-join-build operations is selected for broadcast join distribution by comparing the memory amount to a broadcast memory threshold. The system selects a broadcast join distribution for the subset and a hash-hash join distribution for the remaining hash-join-build operations. The query plan is executed using the broadcast join distribution for the selected subset and the hash-hash join distribution for the remaining operations. This approach optimizes memory usage and join distribution during query execution.
The subject technology receives a first query plan corresponding to a query, the first query plan comprising a set of predicates. The subject technology receives, during execution of a first portion of the first query plan, a set of rowsets. The subject technology determines a set of metrics for a first number of rows from a plurality of rows, the first number of rows corresponding to a first predicate order. The subject technology determines, using a heuristic, a second predicate order based at least in part on the set of metrics. The subject technology processes, during execution of the first portion of the first query plan using the second predicate order, a second set of rowsets, the second set of rowsets comprising a second plurality of rows that correspond to the first portion of the first query plan that has been executed based on the second predicate order.
Provided herein are systems and methods for configuring managed dynamic Iceberg tables. An example method includes parsing, by at least one hardware processor, a table definition to determine a lag duration value, an external volume indicator, and a location indicator. A dynamic table (DT) manager generates a dynamic Iceberg table based on the table definition. The generating is based on selecting an external storage volume of a network-based database system based on the external volume indicator and the location indicator. The DT manager stores a base Iceberg table at a storage location associated with the external storage volume. The DT manager configures the base Iceberg table as the dynamic Iceberg table based on the lag duration value. The lag duration value indicates a maximum time period that a result of a prior refresh of the dynamic Iceberg table lags behind a current time instance.
Various embodiments described herein provide for systems, methods, devices, instructions, and like for swapping artificial intelligence models, such as large language models (LLMs), based on inference request monitoring. In particular, some embodiments monitor inference requests submitted to various inference engines (where each inference engine comprises a group of software containers sharing assigned computing resources) and, based on analysis of inference request data, available models, currently loaded models, or a combination thereof, determine whether to swap out a set of AI models currently active on a select inference engine with another set of AI models available on the select inference engine.
A tag propagator may obtain a SQL statement. As a result of obtaining the SQL statement, object dependencies between objects referenced in the SQL statement may be determined. Tags associated with the determined object dependencies may be further determined. The tags may be propagated.
As described herein, a N-Gram index may be created and the search may be conducted using the index, which will lead to faster search results. The N-Gram index may also include partial N-Gram components to capture more relevant data. Moreover, as described herein, the search may also take into account recent log data that has not yet been indexed. Techniques for building an index store using log data and efficiently searching the index store and log data to process search requests are described herein.
G06F 7/14 - Merging, i.e. combining at least two sets of record carriers each arranged in the same ordered sequence to produce a single set having the same ordered sequence
The subject technology receives a query, the query including a set of statements, the set of statements including a function call, the function call including a declaration of a vector data type as an argument of the function call. The subject technology processes the query, the processing including invoking the function call. The subject technology provides a set of query results from processing the query, the set of query results including a vector data structure corresponding to the vector data type, the vector data structure including a set of elements, each element comprising a numerical data type.
Provided herein are systems and methods for configuring automatic evolution of dynamic tables. An example method includes parsing, by at least one hardware processor, a query associated with a dynamic table to determine a current base object dependency of the dynamic table on at least a first base object. A prior base object dependency of the dynamic table on at least a second base object is retrieved. A delta between data stored by the at least first base object and data stored by the at least second base object is determined. The dynamic table is updated based on the delta.
Embodiments of the present disclosure provide techniques for mountless querying of listing data. A processing device obtains a query that includes a universal listing identifier of a database, wherein the universal listing identifier is different from an identifier for the database. The processing device activates, at runtime, at least one role for accessing the database and shared objects based on the universal listing identifier. The processing device generates, based on the universal listing identifier and the at least one activated role, an in-memory placeholder object associated with the database. The processing device provides access to data of the database based on the in-memory placeholder object and the query.
A data platform is provided. The data platform is configured to receive a request from a client device of a user to run a web application within a computing environment. It initiates an execution of the web application and determines the availability of a cached user interface state of the web application. Upon determining that the cached user interface state is available, the data platform fetches the cached user interface state from the datastore and communicates it to the client device. This allows for displaying an initial user interface to a user by the client device using the cached user interface state while continuing to initialize the web application as the initial user interface is displayed.
G06F 16/957 - Browsing optimisation, e.g. caching or content distillation
G06F 9/451 - Execution arrangements for user interfaces
G06F 16/953 - Querying, e.g. by the use of web search engines
G06F 21/53 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity, buffer overflow or preventing unwanted data erasure by executing in a restricted environment, e.g. sandbox or secure virtual machine
100.
CONFIGURING INTERACTIONS BETWEEN PYTHON AND SQL CELLS IN A NOTEBOOK
Provided herein are systems and methods for configuring interactions between Python and SQL cells in a notebook. An example method includes detecting a run cell message received from a notebook UI application. The run cell message specifies a set of cells of a notebook. At least a first cell of the set of cells is configured as an SQL cell within the notebook. A query within at least one SQL statement associated with the SQL cell is executed to generate cell results. The cell results of the SQL cell are stored in a global namespace of the notebook. Access to the cell results in the global namespace is configured to at least a second cell of the set of cells.
