A cloud asset manager can securely provide multi-tenant access to remote assets while preserving isolation across tenants. The remote asset manager defines various roles for legitimate users of the remote asset manager. The roles are associated with credentials that provide access to the remote assets and/or information about the remote assets maintained by a service provider. And the users map to roles based on attempted actions that access the service provider. Thus, a user's requested action is attempted with credentials associated with a role that maps to the requested action.
Techniques are provided for implementing a distributed control plane to facilitate communication between a container orchestration platform and a distributed storage architecture. The distributed storage architecture hosts worker nodes that manage distributed storage that can be made accessible to applications within the container orchestration platform through the distributed control plane. The distributed control plane includes control plane controllers that are each paired with a single worker node of the distributed storage architecture. Thus, the distributed control plane is configured to selectively route commands to control plane controllers that are paired with worker nodes that are current owners of objects targeted by the commands. In this way, the control plane controllers can facilitate communication and performance of commands between the applications of the container orchestration platform and the worker nodes of the distributed storage architecture.
Systems and methods for creation of bucket-level snapshots and snapshot ownership determination are provided. In one example, a storage system maintains a bucket containing multiple objects each having one or more object versions. A snapshot of the bucket may be efficiently created to protect object versions in the bucket at a specific point in time by simply adding an entry, containing information regarding a snapshot identifier (ID) and a snapshot creation time indicator, to a snapshot metafile. Object-modifying operations may be hooked to internally modify them while making it appear to the client the operation has been successfully completed. For example, before deletion of a particular object, an “Is-Object-Protected” check may be performed based on time indicators of the one or more object versions and respective snapshot creation time indicators. When the particular object is protected, it may be subsequently hidden from the client but maintained as an internal version.
Systems and methods for performing an instant and immediately consistent snapshot restore from a client perspective are provided. In one example, a storage system, may restore a previous version of one or more objects to a bucket based on a snapshot of the bucket by performing a background restore process. During the background restore process, the restoration of the previous version of the one or more objects is made to appear instant to a client. For example, during the background restore process, object accesses by the client associated with a read-only operation may be redirected to content of the snapshot. Additionally or alternatively, during the background restore process, prior to acting on a request from the client involving an object-modifying operation relating to a particular object of the one or more objects, the previous version of the particular object may be restored on-demand.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 16/14 - Détails de la recherche de fichiers basée sur les métadonnées des fichiers
Techniques are provided for an object file system for an object store. Data, maintained by a computing device, is stored into slots of an object. The data within the slots of the object is represented as a data structure comprising a plurality of nodes comprising cloud block numbers used to identify the object and particular slots of the object. A mapping metafile is maintained to map block numbers used to store the data by the computing device to cloud block numbers of nodes representing portion of the data stored within slots of the object. The object is stored into the object store, and the mapping metafile and the data structure are used to provide access through the object file system to portions of data within the object.
G06F 16/14 - Détails de la recherche de fichiers basée sur les métadonnées des fichiers
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 16/11 - Administration des systèmes de fichiers, p. ex. détails de l’archivage ou d’instantanés
The technology disclosed herein enables a higher-level process to perform storage volume management with knowledge of a physical storage backend underlying a storage volume. In a particular example, a method includes mounting a storage volume to a computing node of the computing nodes. The storage volume is stored in a storage pool of a plurality of underlying storage pools. The method further includes determining an identifier for the storage pool, receiving a request to duplicate the storage volume, and determining a second identifier for a second storage pool of the plurality of underlying storage pools to which the storage volume will be duplicated. When the second identifier matches the identifier, creating a clone of the storage volume rather than copying the storage volume to the second storage pool.
Systems and methods for supporting granular snapshots are provided. In one example, a storage system may limit a scope of an operation relating to a snapshot of a bucket by applying a snapshot filter associated with the snapshot in which the snapshot filter specifies one or more criteria for determining a subset of multiple objects of a bucket to which the snapshot applies. In one embodiment, the snapshot filer may represent a prefix specified as part of the operation and application of the snapshot filter may involve filtering the multiple objects based on the prefix. The operation may involve creation of a snapshot, enumeration of objects protected by the snapshot, deletion of the snapshot, or restoration of the snapshot. The association of the snapshot filter with the snapshot may be accomplished by persisting the snapshot filter to a snapshot metafile within a snapshot entry corresponding to the snapshot.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 16/11 - Administration des systèmes de fichiers, p. ex. détails de l’archivage ou d’instantanés
Techniques are provided for directory snapshot and data management. Conventional snapshot functionality creates snapshots at a volume level. Volume level snapshots are inadequate for scale-out storage architectures because a single volume snapshot of a shared storage resource may not satisfy different data protection requirements of clients using the shared storage resource. The disclosed techniques are capable of creating snapshots at a directory level. The directory level snapshots are created and maintained using an inode identity map to track active inode numbers of directory files that have diverged. Snapshot generation numbers are used to determine whether a file is part of a directory for which snapshotting is enabled. A version map used to track versions of a file modified across different directory snapshots and an active file system. A delayed free metafile is used to determine whether file block numbers of a directory can be freed.
Systems and methods are disclosed for implementing a system to generate a knowledge graph of trust relationships between roles in a cloud environment, and to identify misconfigurations that may lead to privilege escalation. In certain embodiments, a method may comprise implementing a graph-based role permission inspection system for identity and access management (IAM) roles in a cloud environment, including generating a graph representation of trust relationships between roles, where a first role having a first set of privileges can endorse a second role having a second set of privileges. The method may further include determining whether the second set of privileges includes a permission not available in the first set of privileges, and generating an indicator that the first role violates a policy when the second set of privileges includes the permission not available in the first set of privileges.
H04L 41/16 - Dispositions pour la maintenance, l’administration ou la gestion des réseaux de commutation de données, p. ex. des réseaux de commutation de paquets en utilisant l'apprentissage automatique ou l'intelligence artificielle
Systems and methods are disclosed for implementing graph-based role inspection for roles in a cloud environment based on a graph neural network (GNN). In certain embodiments, a method may comprise performing graph-based role similarity inspection using a GNN, the graph-based role similarity inspection configured to identify roles, in a graph representation of relationships between identity and access management (IAM) roles of a cloud environment, that are most similar to a target role. The method may include determining a graph structure of the graph representation, identifying the target role, performing a similarly calculation between the target role and other roles in the graph structure to determine similarity scores for the other roles; identifying a similar role having a same security vulnerability as the target role based on the similarity calculation, and correcting the security vulnerability in the similar role based on the identification.
Systems and methods are disclosed for implementing a process for graph database storage optimization, applicable to delta-based cloud asset tracking. In certain embodiments, a method may comprise implementing a delta-based graph storage optimization system for asset tracking in a cloud environment, including storing a graph database representing a configuration of a cloud environment, obtaining configuration settings representing a current state of the cloud environment from a cloud platform, and identifying a delta based on changes between the configuration from the graph database and the configuration settings from the cloud platform. The method may further comprise creating an asset property node based on the delta, and adding the asset property node to the graph database without creating a new graph based on the configuration settings.
Techniques are provided for object store mirroring. Data within a storage tier of a node may be determined as being data to tier out to a primary object store based upon a property of the data. A first object is generated to comprise the data. A second object is generated to comprise the data. The first object is transmitted to the primary data store for storage in parallel with the second object being transmitted to a mirror object store for storage. Tiering of the data is designated as successful once acknowledgements are received from both the primary object that the first object was stored and the mirror object store that the second object was stored.
In one embodiment, a method comprises maintaining state information regarding a data synchronous replication status for a storage object of a primary storage cluster and a replicated storage object of a secondary storage cluster. The method includes temporarily disallowing input/output (I/O) operations for the storage object when the storage object of the primary storage cluster has a failure, which causes an internal state as out of sync for the storage object while maintaining an external state as in sync for external entities. The method performs persistent inflight tracking and reconciliation of I/O operations with a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster and performs a resynchronization between the storage object and the replicated storage object based on the persistent inflight tracking and reconciliation of I/O operations.
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
14.
METHODS AND SYSTEMS TO REDUCE LATENCY OF INPUT/OUTPUT (I/O) OPERATIONS BASED ON FILE SYSTEM OPTIMIZATIONS DURING CREATION OF COMMON SNAPSHOTS FOR SYNCHRONOUS REPLICATED DATASETS OF A PRIMARY COPY OF DATA AT A PRIMARY STORAGE SYSTEM TO A MIRROR COPY OF THE DATA AT A CROSS-SITE SECONDARY STORAGE SYSTEM
Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshot process for storage objects. A computer-implemented method comprises performing a baseline transfer from at least one storage object of a first storage node to at least one replicated storage object of a second storage node, starting the common snapshot process including stop processing of I/O operations, performing a snapshot create operation on the primary storage site for the at least one storage object of the first storage node, resuming processing of I/O operations, and assigning a new universal unique identifier (UUID) to the at least one storage object of the second storage node after resuming processing of I/O operations with the new UUID to identify when file system contents are different than the baseline transfer.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
G06F 16/11 - Administration des systèmes de fichiers, p. ex. détails de l’archivage ou d’instantanés
G06F 16/178 - Techniques de synchronisation des fichiers dans les systèmes de fichiers
15.
SLICE FILE RECOVERY USING DEAD REPLICA SLICE FILES
Techniques are provided for repairing a primary slice file, affected by a storage device error, by using one or more dead replica slice files. The primary slice file is used by a node of a distributed storage architecture as an indirection layer between storage containers (e.g., a volume or LUN) and physical storage where data is physically stored. To improve resiliency of the distributed storage architecture, changes to the primary slice file are replicated to replica slice files hosted by other nodes. If a replica slice file falls out of sync with the primary slice file, then the replica slice file is considered dead (out of sync) and could potentially comprise stale data. If a storage device error affects blocks storing data of the primary slice file, then the techniques provided herein can repair the primary slice file using non-stale data from one or more dead replica slice files.
Data is replicated on a backup node, where the granularity of the replication can be less than a full volume. A data consistency group comprising a subset of data for a volume is defined for a primary node. A set of differences for the data consistency group is sent to a backup node. The backup node creates change logs in response to receiving the set of differences. In response to receiving a request to access a file having data in the data consistency group, the backup node creates a clone of the file. The backup node determines whether an update to a data block of the file exists in the change logs. In response to determining that the update to the data block exists in the change logs, the backup node updates a copy of the data block for the cloned file with data in the change logs.
G06F 16/27 - Réplication, distribution ou synchronisation de données entre bases de données ou dans un système de bases de données distribuéesArchitectures de systèmes de bases de données distribuées à cet effet
17.
VIRTUAL MACHINE BACKUP FROM COMPUTING ENVIRONMENT TO STORAGE ENVIRONMENT
Techniques are provided for backing up virtual machines from a computing environment to a storage environment. A virtual machine agent is utilized to generate a snapshot of the virtual machine. Metadata comprising a snapshot identifier of the snapshot and virtual disk information of virtual disks captured by snapshot is generated at the computing environment. The metadata is retrieved and used to create a metafile that is transferred to the storage environment within which snapshots of the virtual machine are to be stored. The snapshot is retrieved from the computing environment and is packaged into a snapshot package having a protocol format used by the storage environment. The snapshot package is transferred to the storage environment.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
G06F 9/455 - ÉmulationInterprétationSimulation de logiciel, p. ex. virtualisation ou émulation des moteurs d’exécution d’applications ou de systèmes d’exploitation
18.
METHODS AND SYSTEMS TO REDUCE LATENCY OF INPUT/OUTPUT (I/O) OPERATIONS BASED ON CONSISTENCY POINT OPTIMIZATIONS DURING CREATION OF COMMON SNAPSHOTS FOR SYNCHRONOUS REPLICATED DATASETS OF A PRIMARY COPY OF DATA AT A PRIMARY STORAGE SYSTEM TO A MIRROR COPY OF THE DATA AT A CROSS-SITE SECONDARY STORAGE SYSTEM
Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshotprocedure for storage objects. A computer-implemented method includes initiating a snap create handler operation for a storage object of a batch of storage objects having a plurality of replicated datasets with each replicated dataset having a synchronous replication relationship between at least one storage object of the first storage node and at least one replicated storage object of the second storage node, determining whether a consistency point is currently in progress or not, and providing a hint to accelerate a currently in progress consistency point when the consistency point is currently in progress.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
G06F 16/11 - Administration des systèmes de fichiers, p. ex. détails de l’archivage ou d’instantanés
G06F 16/178 - Techniques de synchronisation des fichiers dans les systèmes de fichiers
19.
TRANSITIONING VOLUMES BETWEEN STORAGE VIRTUAL MACHINES
A volume rehost tool migrates a storage volume from a source virtual server within a distributed storage system to a destination storage server within the distributed storage system. The volume rehost tool can prevent client access to data on the volume through the source virtual server until the volume has been migrated to the destination virtual server. The tool identifies a set of storage objects associated with the volume, removes configuration information for the set of storage objects, and removes a volume record associated with the source virtual server for the volume. The tool can then create a new volume record associated with the destination virtual server, apply the configuration information for the set of storage objects to the destination virtual server, and allow client access to the data on the volume through the destination virtual server.
G06F 9/455 - ÉmulationInterprétationSimulation de logiciel, p. ex. virtualisation ou émulation des moteurs d’exécution d’applications ou de systèmes d’exploitation
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
Approaches for providing a non-disruptive file move are disclosed. A request to move a target file from the first constituent to the second constituent is received. The file has an associated file handle. The target file in the first constituent is converted to a multipart file in the first constituent with a file location for the new file in the first constituent. A new file is created in the second constituent. Contents of the target file are moved to a new file on the second constituent while maintaining access via the associated file handle via access to the multipart file. The target file is deleted from the first constituent.
VERIFICATION OF A PUBLISHED IMAGE HAVING A PREDEFINED PORTION THAT HAS BEEN ALTERED BY A CLOUD PROVIDER PRIOR TO BEING MADE AVAILABLE VIA A MARKETPLACE OF THE CLOUD PROVIDER
Systems and methods for verifying an executable portion of a published cloud image represents an unaltered version of an executable portion of a corresponding original cloud image are provided. In one embodiment, modification of a predefined portion of a cloud image by a cloud provider prior to its publication via a marketplace of the cloud provider is proactively addressed as part of (i) an automated signing process performed by a software publisher on the original cloud image prior to delivery to the cloud provider and (ii) a corresponding background verification process performed on the published cloud image on behalf of users by a management platform. The signing and verification processes are operable to exclude the predefined portion when creating their respective digests, thereby allowing the signed digest created prior to the modification to remain useful as part of a subsequent digest comparison performed by the verification process.
H04L 9/32 - Dispositions pour les communications secrètes ou protégéesProtocoles réseaux de sécurité comprenant des moyens pour vérifier l'identité ou l'autorisation d'un utilisateur du système
G06F 8/71 - Gestion de versions Gestion de configuration
H04L 9/30 - Clé publique, c.-à-d. l'algorithme de chiffrement étant impossible à inverser par ordinateur et les clés de chiffrement des utilisateurs n'exigeant pas le secret
22.
Cloud Based Interface for Protecting and Managing Data Stored in Networked Storage Systems
Methods and systems for managing storage is provided. One method includes initializing, a storage service user interface (“SSUI”) within a cloud manager user interface (“CMUI”), the SSUI enables a storage service operation selected from a backup operation, a restore operation and a cloning operation associated with a storage object stored at a cloud volume presented by a cloud layer, the CMUI presented by a cloud provider; transmitting, by the SSUI, an authorization token to a Software As a Service (“SaaS”) layer for authenticating deployment of the SSUI; upon validating the authorization token, initializing a SSUI agent to interface with a deployed storage micro-service layer offered by the cloud layer to execute the storage service operation; transmitting, by the SSUI, an application programming interface (API) request for the SSUI agent for executing the storage service operation; and executing, by the deployed storage micro-service layer, the storage service operation.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 21/44 - Authentification de programme ou de dispositif
H04L 67/1097 - Protocoles dans lesquels une application est distribuée parmi les nœuds du réseau pour le stockage distribué de données dans des réseaux, p. ex. dispositions de transport pour le système de fichiers réseau [NFS], réseaux de stockage [SAN] ou stockage en réseau [NAS]
H04L 67/51 - Découverte ou gestion de ceux-ci, p. ex. protocole de localisation de service [SLP] ou services du Web
23.
REDUCING POTENTIAL DATA-LOSS SCENARIOS WHEN USING EPHEMERAL STORAGE AS BACKING STORAGE FOR JOURNALING BY A VIRTUAL STORAGE SYSTEM
Systems and methods for flushing an operation log journal to both ephemeral storage and persistent storage during a shutdown sequence of a virtual storage system to minimize data-loss scenarios are provided. According to one embodiment, the shutdown or reboot scenarios that result in loss of data are minimized by using persistent storage as a backup to ephemeral storage when the scenario results in rehosting of virtual storage system. For example, responsive to an event indicative of an imminent shutdown or reboot of the virtual storage system, vNVRAM memory may be flushed to both ephemeral storage and persistent storage (e.g., a boot disk). In this manner, when the virtual storage system is rehosted after an unplanned shutdown or reboot resulting from an unrecoverable host error (other than an unrecoverable hardware failure), the operation log journal may be recovered from persistent storage to facilitate vNVRAM replay and avoid data loss.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
G06F 9/455 - ÉmulationInterprétationSimulation de logiciel, p. ex. virtualisation ou émulation des moteurs d’exécution d’applications ou de systèmes d’exploitation
Techniques are provided for journal replay optimization. A distributed storage architecture can implement a journal within memory for logging write operations into log records. Latency of executing the write operations is improved because the write operations can be responded back to clients as complete once logged within the journal without having to store the data to higher latency disk storage. If there is a failure, then a replay process is performed to replay the write operations logged within the journal in order to bring a file system up-to-date. The time to complete the replay of the write operations is significantly reduced by caching metadata (e.g., indirect blocks, checksums, buftree identifiers, file block numbers, and consistency point counts) directly into log records. Replay can quickly access this metadata for replaying the write operations because the metadata does not need to be retrieved from the higher latency disk storage into memory.
Techniques are provided for implementing garbage collection and bin synchronization for a distributed storage architecture of worker nodes managing distributed storage composed of bins of blocks. As the distributed storage architecture scales out to accommodate more storage and worker nodes, garbage collection used to free unused blocks becomes unmanageable and slow. Accordingly garbage collection is improved by utilizing heuristics to dynamically speed up or down garbage collection and set sizes for subsets of a bin to process instead of the entire bin. This ensures that garbage collection does not use stale information about what blocks are in-use, and ensures garbage collection does not unduly impact client I/O processing or conversely falls behind on garbage collection. Garbage collection can be incorporated into a bin sync process to improve the efficiency of the bin sync process so that unused blocks are not needlessly copied by the bin sync process.
Systems and methods for using object storage as a primary storage tier for a file system of a virtual storage system are provided. According to one embodiment, ephemeral storage of a host on which a virtual storage system is running in the cloud is used as an external cache in which all or some subset of file system metadata may be stored, for example, to boost read IOPS and reduce latency of the read path. In various examples, external cache policies may be defined to trigger adding to, updating, and/or re-warming the external cache to accelerate access to file system metadata, thereby making a virtual storage solution that maintains all file system data and file system metadata within object storage a suitable and cost-effective platform for certain types of workloads. In some example, the working set of a workload may also be kept in the external cache to further accelerate read performance of the virtual storage system.
Systems and methods for making instance-type recommendations are provided. In various examples, an instance type recommendation system (internal or external to a cloud) provides users (cloud customers) with instance type recommendations and may automatically adjust their instance type groups (ITGs). The instance type recommendations may take into consideration other users with similar requirements and/or be based on frequency of co-occurrence of an instance type of the user at issue with one or more other instance types used by other users as reflected by their respective current ITGs. For example, a multi-layer perceptron (MLP) neural network may be trained by breaking instance types down into respective attributes and causing the MLP to encode the attributes as features and the training may make use of a triplet loss function that minimizes a distance between an anchor and a positive input while maximizing a distance between the anchor and a negative input.
Techniques are provided for performing a storage operation targeting objects stored across multiple storage tiers of a cloud storage environment. A volume may be backed up as objects stored across the multiple storage tiers of the cloud storage environment, such as a standard storage tier directly accessible to the storage operation, an archival storage tier not directly accessible to the storage operation, etc. The storage operation may target the objects, such as where the storage operation is a directory restore operation to restore a directory of the volume. The storage operation can be successfully implemented such as to restore the directory even though objects of the storage operation are stored across the multiple storage tiers of the cloud storage environment.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
29.
KEY VALUE STORE WITH DISTRIBUTED SHARED ACCESS BY CLIENTS
Techniques are provided for hosting a key value store. A persistent storage backend is used to centrally host a key value store as disaggregated storage shared with a plurality of clients over a network fabric. A network storage appliance is connected to the plurality of clients over the network fabric, and is configured with a key value store interface. The key value store interface is configured to receive a key value command from a client. The key value store interface parses the key value command to identify a translation layer binding for a key value store targeted by the key value command. The key value store interface translates the key value command into a key value operation using the translation layer binding, and executes the key value operation upon the key value store.
Methods and systems involve sequentially storing compressed data segments received at a storage system in disk blocks such that at least one of the disk blocks includes compressed data from at least two of the compressed data segments.
Systems, methods, and software are disclosed herein for phased-in restoration of an application hosted on a cloud orchestration platform in various implementations. In an implementation, a computing apparatus receives a configuration for a multiphase restoration process for restoring resources of an application to a destination platform, the restoration occurring in phases. To implement the multiphase restoration process, the computing apparatus captures a backup of application data of the application, then restores a phase including selected resources of the application to the destination platform based on the backup and according to the configuration. The computing apparatus validates the selected resources at the destination platform, then restores a next phase to the destination platform based on the backup and according to the configuration.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
32.
MANAGEMENT OF DATA STAGING FOR WRITING TO OBJECT BASED STORAGE
A method, computing device, and non-transitory machine-readable medium for managing incoming/outgoing flow through a staging area. A request to write data to an object based storage is received. A determination is made that the data is to be first staged within a transfer data structure. The transfer data structure is in a first storage tier and the object based storage is in a second storage tier. A determination is made that an amount of storage space used in the transfer data structure exceeds a start throttle threshold. In response to the determination that the amount of storage space exceeds the start throttle threshold, the request is sent into a queue selected from a set of throttling queues, wherein requests in the queue are handled in a first in, first out (FIFO) order. The request in the queue is processed based on the FIFO order and a dynamic throttle rate.
Disclosed herein are methods and systems for the operation of a resource management service. The resource management service deploys reclaimable compute instances from a resource pool and continuously generates predicted remaining lifespans for the deployed reclaimable compute instances. The predicted remaining lifespan is monitored to determine if the predicted remaining lifespan is below a threshold value. In response to the predicted remaining lifespan for a reclaimable compute instance falling below a threshold value, the resource management service instructs the reclaimable compute instance to create an application state snapshot of an application running thereon. A subsequent compute instance is deployed from the resource pool, on which the application can be restored to a previous state using the application state snapshot.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 9/455 - ÉmulationInterprétationSimulation de logiciel, p. ex. virtualisation ou émulation des moteurs d’exécution d’applications ou de systèmes d’exploitation
34.
Predictive Scale-Up Of Compute Nodes In A Software Orchestration Cluster
The disclosure describes a node management service that proactively scales up compute nodes in a compute cluster. The node management service interfaces with an orchestration service, a compute provider and a compute cluster running instances of an object. The node management service receives meta data from an orchestration service indicating the desired number of instances of an object. Based on the desired number of instances, the node management service obtains, from the compute provider, new compute nodes for the compute cluster to accommodate the desired number of instances.
Disclosed herein are methods and systems for the operation of a resource management service. The resource management service deploys reclaimable compute instances from a resource pool and continuously generates predicted remaining lifespans for the deployed reclaimable compute instances. The predicted remaining lifespan is monitored to determine if the predicted remaining lifespan is below a threshold value. In response to the predicted remaining lifespan for a reclaimable compute instance falling below a threshold value, the resource management service instructs the reclaimable compute instance to create an application state snapshot of an application running thereon. A subsequent compute instance is deployed from the resource pool, on which the application can be restored to a previous state using the application state snapshot.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
36.
ANOMALY DETECTION BASED ON STORAGE PROTOCOL CONNECTIONS
Systems and methods for anomaly detection within a storage system based on a number of storage protocol connections are provided. In one example, alerting thresholds based on storage protocol connections are made more relevant and customer-specific by taking into consideration, among other things, one or more of the type/class/model of storage system, the configuration of the storage system, the type of the workload making use of the storage system, and various windows of time. Based on the customer-specific nature of the alerts generated by the proposed alerting system, potential performance issues being experienced by the storage system may be more efficiently identified and remediated, for example, by customer service/support, the customer, or possibly by automated storage system (on-box) functionality.
A method, computing device, and non-transitory machine-readable medium for implementing a client-controllable bypass mechanism for directly writing to object based storage. In one or more embodiments, a write request for writing data to a volume is received from a client in which the volume represents both first storage within a first storage tier and second storage within a second storage tier. A determination is made that a bypass write mode is enabled for the volume in which the bypass write mode allows directly assigning the data to the second storage tier. A set of objects is built for the data in which the data is stored in a transfer data structure in the first storage tier. The set of objects, which includes the data in the transfer data structure, is sent to the second storage tier.
Disclosed herein are systems, methods, and software for the operation of a ransomware detection system. The ransomware detection system generates a decoy file based on characteristics of an existing file in a file system. The decoy file is effectively indistinguishable from the existing file from the perspective of the ransomware but contains simulated data rather than authentic data. The ransomware detection system identifies a location in the file system and deploys the decoy file to the location. The decoy is then monitored to detect changes by comparing a ground truth for the decoy file to the current state of the decoy file. The decoy file is checked for changes at a rate associated with the identified location. Where a change is detected, an alert is sent to a ransomware mitigation process, which initiates ransomware mitigation.
The disclosure describes a data protection service that generates semantic descriptions of protected data volumes. The data protection service queries a monitoring service with the generated semantic descriptions. The monitoring service responds to the queries with indications of whether and data items on the dark web match the semantic descriptions. When a query receives a positive response from the monitoring service, the data protection service iteratively refines the semantic description and queries the monitoring service with the refined semantic descriptions until a breach is detected. Once a breach is detected, the data protection service initiates a mitigation action.
The disclosure describes a data protection service that generates semantic descriptions of protected data volumes. The data protection service queries a monitoring service with the generated semantic descriptions. The monitoring service responds to the queries with indications of whether and data items on the dark web match the semantic descriptions. When a query receives a positive response from the monitoring service, the data protection service iteratively refines the semantic description and queries the monitoring service with the refined semantic descriptions until a breach is detected. Once a breach is detected, the data protection service initiates a mitigation action.
In one embodiment, a computer implemented method includes comprises storing objects in a first bucket and files in a second bucket of a first storage cluster of the distributed storage system, initiating an audit job on the first storage cluster, synchronously replicating audit configuration data and mirroring audit data (e.g., audit files, logs) from the first storage cluster to the second storage cluster, performing a switchover process from the first storage cluster to the second storage cluster, and initiating an audit job on the second storage cluster based on the audit configuration during the switchover process. The first storage cluster initially handles input/output operations for a software application before the switchover process.
G06F 16/27 - Réplication, distribution ou synchronisation de données entre bases de données ou dans un système de bases de données distribuéesArchitectures de systèmes de bases de données distribuées à cet effet
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
Techniques are provided for rebuilding a destination file system as part of a resync operation. A file system of a first node may be replicated to a second node as a destination file system. The file system may include clone files that were created as clones of parent files. The clone files may have been created in a storage efficient manner where a clone file refers to data of a parent file, instead physically storing duplicate instances of the parent data of the parent file. The techniques provided herein are capable of replicating the clone files to the second node while preserve the storage efficiency of the clone files during a resync operation that rebuilds the destination file system after recovering from a failure of the second node.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 11/20 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel en utilisant un masquage actif du défaut, p. ex. en déconnectant les éléments défaillants ou en insérant des éléments de rechange
43.
Resource Distribution Engine(s) For Allocating And Securing Reclaimable Resources Within A Cloud Environment
Various embodiments of the present technology generally relate to systems and methods for providing a resource distribution engine. In an example, a method includes identifying, by an instance management service, a request from a cluster orchestrator to acquire computing resources associated with deployment of an application. The method may also include determining, by the instance management service, markets containing reclaimable resources suitable for the computing resources, and performing, by a resource distribution engine of the instance management service, a resource distribution process to determine a resource distribution of the reclaimable resources within the markets. The resource distribution of the reclaimable resources may include a distribution of instance type of the reclaimable resources across the markets. The method may also include securing, by the instance management service, the reclaimable resources identified by the resource distribution as provisioned reclaimable resources.
Techniques are provided for replicating clone files from a source file system to a destination file system in a computing system. A file system of a first storage node may be replicated to a second storage node such as for data protection and backup purposes. The file system may include clone files that were created as clones of parent files. The clone files may have been created in a storage efficient manner where a clone file refers to data of a parent file, instead physically storing duplicate instances of the parent data of the parent file. The techniques provided herein are capable of replicating the clone files to the second node while preserve the storage efficiency of the clone files.
One or more techniques and/or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and/or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and/or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and/or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and/or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.
Techniques are provided for a get key bot that securely provides access to passphrases. A set key workflow is executed to generate a get key executable binary that is implemented as the get key bot. The get key executable binary is encapsulated with encrypted information that includes a verified user identifier, a passphrase, and/or a bot expiry time. Upon receiving a request for the passphrase from a requestor, the get key executable binary is invoked. The encrypted information is decrypted and compared to a logged-in user identifier and current time for verification. In response to successful verification, the passphrase is provided to the requestor. Otherwise, the requestor is denied access to the passphrase.
Methods and systems use a central validation module of a microservice-based system to interface an external identity provider (IDP) among a plurality of external IDPs to receive authentication and/or authorization information for an external request from a client. After the authentication and/or authorization information has been received for the external request, an internal request containing the authentication and/or authorization information is generated by the central validation module, which is transmitted to at least one microservice of the microservice-based system to provide services in response to the external request. The internal request is validated with the central validation module by each of the at least one microservice, which executes at least one operation to provide the services. A response that is based at least partly on results of the services provided by the at least one microservice is returned for the external request to the client.
G06F 15/16 - Associations de plusieurs calculateurs numériques comportant chacun au moins une unité arithmétique, une unité programme et un registre, p. ex. pour le traitement simultané de plusieurs programmes
H04L 67/1097 - Protocoles dans lesquels une application est distribuée parmi les nœuds du réseau pour le stockage distribué de données dans des réseaux, p. ex. dispositions de transport pour le système de fichiers réseau [NFS], réseaux de stockage [SAN] ou stockage en réseau [NAS]
48.
Managing Object Lock Settings During Cross-Grid Replication Within A Distributed Storage System
Various embodiments of the present technology generally relate to systems and methods for providing managing object lock settings during cross-grid replication within distributed storage systems. In an example, ingestion of an object into a first grid of a distributed storage system may be detected. Responsive to detecting ingestion of the object, object lock settings for the object may be determined. Once the object lock settings are determined, the object lock settings may be validated against destination object lock settings. If the destination object lock settings are validated, cross-grid replication of the object may be initiated. During cross-grid replication, the object lock header may be provided in a replication payload transmitted from the first grid to a second grid. When the object is replicated, the destination object lock settings may be determined for the object, which may include the object lock settings as identified in the object lock header.
Systems and methods for reducing the provisioned storage capacity of a disk or aggregate of disks of a storage appliance while the storage appliance continues to serve clients are provided. According to one embodiment, the size of the aggregate may be reduced by shrinking the file system of the storage appliance and removing a selected disk from the aggregate. When an identified shrink region includes the entire addressable PVBN space of the selected disk, the file system may be shrunk by relocating valid data from the selected disk elsewhere within the aggregate. After the valid data is relocated, the selected disk may be removed from the aggregate, thereby reducing the provisioned storage capacity of the aggregate by the size of the selected disk.
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
50.
NETWORK ADDRESS MIGRATION BETWEEN DIFFERENT NETWORKS BY UPDATING NETWORK CONFIGURATION USING A DESTINATION COMPUTE INSTANCE IN A CLOUD ENVIRONMENT TO REDUCE DISRUPTIONS
Network address migration using a destination compute instance to update network configuration information in a cloud environment is disclosed. A network interface either using a private address within a subnet corresponding to the created network interface or using a floating address outside of the subnet corresponding to the created network interface is created. A first node of the HA pair with a service provider interface. The first node is an active data server of the HA pair, and the second node is a backup node of the HA pair. Requests are serviced the first node using the created network interface. Upon failure of the first node, the second node performs a failover, wherein if the first node was utilizing a floating address, the second node registers the second node with the service provider interface by adding an address of the second node to the route table.
H04L 61/2539 - Traduction d'adresses de protocole Internet [IP] en masquant les adressesTraduction d'adresses de protocole Internet [IP] en gardant les adresses anonymes
H04L 67/10 - Protocoles dans lesquels une application est distribuée parmi les nœuds du réseau
NETWORK ADDRESS MIGRATION BETWEEN DIFFERENT NETWORKS BY UPDATING FLOATING LOGICAL NETWORK INTERFACES USING A DESTINATION COMPUTE INSTANCE IN A CLOUD ENVIRONMENT TO REDUCE DISRUPTIONS
Network address migration using a destination compute instance to update network configuration information in a cloud environment is disclosed. A network interface either using a private address within a subnet corresponding to the created network interface or using a floating address outside of the subnet corresponding to the created network interface is created. A first node of the HA pair with a service provider interface. The first node is an active data server of the HA pair and the second node is a backup node of the HA pair. Requests are serviced the first node using the created network interface. Upon failure of the first node, the second node performs a failover, wherein if the first node was utilizing a floating address, the second node registers the second node with the service provider interface by adding an address of the second node to the route table.
Systems and methods for performing an online migration of a storage solution deployment from one disk type to another are provided. In various examples described herein, after a storage solution has been deployed that utilizes disks of a first disk type (e.g., a Google Cloud Platform (GCP) Persistent Disk (PD)), the storage solution deployment may be modified to make use of a second disk type (e.g., a GCP Hyperdisk (HD)) without incurring downtime. In one embodiment, the online migration involves adding new disks of the second disk type to an aggregate of which the original disks of the first disk type are a part, mirroring writes to both the original disks and the new disks, and copying valid data from a given original disk to a corresponding new disk during a background scan of each of the set of original disks.
Techniques are provided for key-value store and file system integration to optimize key value store operations. A key-value store is integrated within a file system of a node. A log structured merge tree of the key-value store may be populated with a key corresponding to a content hash of a value data item stored separate from the key. A random distribution search may be performed upon a sorted log of the log structured merge tree to identify the key for accessing the value data item. A starting location for the random distribution search is derived from key information, a log size of the sorted log, and/or a keyspace size of a keyspace associated with the key.
G06F 16/22 - IndexationStructures de données à cet effetStructures de stockage
G06F 16/21 - Conception, administration ou maintenance des bases de données
G06F 16/215 - Amélioration de la qualité des donnéesNettoyage des données, p. ex. déduplication, suppression des entrées non valides ou correction des erreurs typographiques
Various embodiments of the present technology generally relate to systems and methods for providing cross-grid replication within distributed storage systems. In an example, a method includes identifying an object for ingest into a first storage grid containing a first distributed storage system and replicating the object to one or more nodes within the first storage grid. The method may also include determining a cross-grid replication status of the object to a second storage grid containing a second distributed storage system and performing a cross-grid replication of the object to the second storage grid based on the cross-grid replication status of the object.
G06F 16/27 - Réplication, distribution ou synchronisation de données entre bases de données ou dans un système de bases de données distribuéesArchitectures de systèmes de bases de données distribuées à cet effet
Method and systems for co-locating journaling and data storage based on write requests are provided. In one example, a first logical storage unit for storing write operation records is provided by a cluster of multiple nodes representing a distributed storage system. The first logical storage unit is divided into a volume partition and a journal partition that includes a first log and a second log. A client write request including metadata and data is received by a first node of the cluster. The metadata is recorded in a first location in an active log of the first log and the second log and the data is recorded in a second location in the active log during a single input/output (I/O) operation performed by the first node. A reply is sent by the first node to the client after the metadata and the data are recorded in the journal partition.
Techniques are provided for performing adaptive sampling for data summarization. An insight service may provide monitoring, troubleshooting, optimization, security, and/or other functionality for a computing environment. The insight service may intake millions to billions of events on a monthly basis from the computing environment, which are stored within a database. The insight service may provide data summarization for the events, which may include access patterns (e.g., file access patterns), anomalies, and ransomware detection. Dynamically querying and generating the data summarization may be impractical due to the sheer amount of events. Accordingly, adaptive sampling is provided for merely sampling certain events based upon various thresholds and criteria being met so that an evaluation output can be dynamically and efficiently generated within an acceptable time as the data summarization.
Systems, methods, and software are disclosed herein for detecting encrypted data in various implementations. In an implementation, a computing apparatus determines byte frequency distribution values associated with a compute workload. The computing apparatus executes a machine learning model trained to differentiate between encrypted portions and non-encrypted portions of the compute workload based on the byte frequency distribution values. The computing apparatus monitors an encrypted share of the compute workload represented by the encrypted portions and, in response to the encrypted share meeting or exceeding a threshold, initiating a mitigative action.
Methods and systems for a networked storage system are provided. One method includes predicting an IOPS limit for a plurality of storage pools based on a maximum allowed latency of each storage pool, the maximum allowed latency determined from a relationship between the retrieved latency and a total number of IOPS from a resource data structure; identifying a storage pool whose utilization has reached a threshold value, the utilization based on a total number of IOPS directed towards the storage pool and a predicted IOPS limit; detecting a bully workload based on a numerical value determined from a total number of IOPS issued by the bully workload for the storage pool and a rising step function; and implementing a corrective action to reduce an impact of the bully workload on a victim workload.
The technology disclosed herein enables a storage orchestrator controller to prevent residual data from being written to a storage volume when a node fails non-gracefully. In a particular example, a method includes determining a health status of nodes in the cluster and, in response to determining a node in the cluster failed, marking the node as dirty. After marking the node as dirty and in response to determining the node is ready, the method includes directing the node to erase data in one or more write buffers at the node. The one of more write buffers buffer data for writing to one or more storage volumes when the one or more storage volumes are mounted by the node. After the one or more write buffers are erased, the method includes marking the node as clean.
G06F 11/00 - Détection d'erreursCorrection d'erreursContrôle de fonctionnement
G06F 11/16 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel
G06F 11/18 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel en utilisant un masquage passif du défaut des circuits redondants, p. ex. par logique combinatoire des circuits redondants, par circuits à décision majoritaire
G06F 11/20 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel en utilisant un masquage actif du défaut, p. ex. en déconnectant les éléments défaillants ou en insérant des éléments de rechange
H04L 65/00 - Dispositions, protocoles ou services dans les réseaux de communication de paquets de données pour prendre en charge les applications en temps réel
H04L 67/00 - Dispositions ou protocoles de réseau pour la prise en charge de services ou d'applications réseau
60.
DISTRIBUTED CONTROL PLANE FOR HANDLING WORKER NODE FAILURES OF A DISTRIBUTED STORAGE ARCHITECTURE
Techniques are provided for implementing a distributed control plane to facilitate communication between a container orchestration platform and a distributed storage architecture. The distributed storage architecture hosts worker nodes that manage distributed storage that can be made accessible to applications within the container orchestration platform through the distributed control plane. The distributed control plane includes control plane controllers that are each paired with a single worker node of the distributed storage architecture. The distributed control plane is configured to selectively route commands to control plane controllers that are paired with worker nodes that are current owners of objects targeted by the commands. If a worker node fails and ownership of an object has changed from the failed worker node to another worker node, then subsequent commands are re-routed to a control plane controller paired with the other worker node now owning the object in place of the failed worker node.
Systems and methods are described for use of dynamically extensible file system (DEFS) labels to facilitate performance of disaggregated storage system workflows. In various examples, DEFS labels provide an efficient mechanism through which disaggregated workflows (e.g., sub-workflows of a cluster-wide workflows relating to respective DEFS) may inform each other regarding their respective current states. For example, various flags may be maintained within a DEFS label of a given DEFS indicative of, among other things, whether the DEFS is corrupted, whether the DEFS is online or offline, whether a file system consistency check is in process for the DEFS, etc. During performance of a cluster-wide workflow, the various individual disaggregated workflows that are required to carry out the cluster-wide workflow may coordinate and/or otherwise synchronize their activities with reference to the DEFS state information maintained within the respective DEFS labels.
Systems and methods are provided for implementation and use of disaggregated storage by a distributed storage system. In various examples described herein, the association of blocks of storage space within a storage pod with a given dynamically extensible file system (DEFS) may be in large chunks of one or more gigabytes (GB), which may be referred to herein as “allocation areas” (AAs) that each include multiple RAID stripes. The use of large, multi-GB chunks, as the unit of space allocation/assignment to DEFSs of a distributed storage system facilitates ease of management and independence of write allocation. The use of AAs described herein further allows disk space to be used more fluidly across individual (nodes) storage systems of a distributed storage system (cluster of nodes working together), thereby eliminating silos of storage; and processing resource (e.g., CPU) load may be distributed across the cluster.
Systems and methods for transferring ownership of allocation areas (AA) and associated metadata information to facilitate space balancing within a distributed storage system that makes use of disaggregated storage are provided. In various examples described herein, the unit of storage space assignment used to associate blocks of storage space within a storage pod with a given dynamically extensible file system (DEFS) is an AA, representing a large chunk of one or more gigabytes (GB). The use of AAs described herein allows disk space to be used more elastically across individual nodes of a storage cluster. When partial AAs are transferred among DEFSs, the transfer of associated metadata information may make use of various optimizations to minimize the amount of data copying performed. For example, copying physical volume block numbers (PVBNs) directly or indirectly indicative of locations of metafile data is faster than copying the underlying metafile data.
Systems and methods are provided for implementation and use of disaggregated storage by a distributed storage system. In various examples described herein, the association of blocks of storage space within a storage pod with a given dynamically extensible file system (DEFS) may be in large chunks of one or more gigabytes (GB), which may be referred to herein as "allocation areas" (AAs) that each include multiple RAID stripes. The use of large, multi-GB chunks, as the unit of space allocation/assignment to DEFSs of a distributed storage system facilitates ease of management and independence of write allocation. The use of AAs described herein further allows storage device space to be used more fluidly across individual (nodes) storage systems of a distributed storage system (cluster of nodes working together), thereby eliminating silos of storage; and processing resource (e.g., CPU) load may be distributed across the cluster.
Systems and methods for implementing context checks that account for the potential for file movement across nodes of a cluster of a distributed storage system are provided. Context data utilized for performing context checking in connection with performing read operations may include a buffer tree identifier (bufftree ID), a data ID, and an epoch in which the bufftree ID represents a volume ID that is unique across the cluster, the data ID corresponds to a file block number within the file at issue from which data is being read, and the epoch is a value that facilitates cluster-wide timeline checks. In one embodiment, the bufftree ID may be ensured to be unique across the cluster by, during the process of creating a new volume, combining a unique ID of the DEFS hosting the new volume with a monotonically increasing volume count maintained for the DEFS.
Systems and methods are provided for implementation and use of disaggregated storage by a distributed storage system. In various examples described herein, the association of blocks of storage space within a storage pod with a given dynamically extensible file system (DEFS) may be in large chunks of one or more gigabytes (GB), which may be referred to herein as “allocation areas” (AAs) that each include multiple RAID stripes. The use of large, multi-GB chunks, as the unit of space allocation/assignment to DEFSs of a distributed storage system facilitates ease of management and independence of write allocation. The use of AAs described herein further allows disk space to be used more fluidly across individual (nodes) storage systems of a distributed storage system (cluster of nodes working together), thereby eliminating silos of storage; and processing resource (e.g., CPU) load may be distributed across the cluster.
Systems and methods are provided for implementation and use of disaggregated storage by a distributed storage system. In various examples described herein, the association of blocks of storage space within a storage pod with a given dynamically extensible file system (DEFS) may be in large chunks of one or more gigabytes (GB), which may be referred to herein as “allocation areas” (AAs) that each include multiple RAID stripes. The use of large, multi-GB chunks, as the unit of space allocation/assignment to DEFSs of a distributed storage system facilitates ease of management and independence of write allocation. The use of AAs described herein further allows disk space to be used more fluidly across individual (nodes) storage systems of a distributed storage system (cluster of nodes working together), thereby eliminating silos of storage; and processing resource (e.g., CPU) load may be distributed across the cluster.
Managing access to stored objects includes receiving, by a storage OS in a computing system, an access key identifier (ID) and access key associated with a user of an object storage service (OSS); determining, by a directory service, whether the user is a member of a group authorized to access a storage object in at least one of a network accessible storage (NAS) volume and an OSS bucket, based at least in part on the access key ID; in response to the user being a group member, attempting to authenticate the user by the storage OS with the directory service based at least on the access key ID and the access key; and in response to the user being authenticated by the directory service, allowing, by the storage OS, access by the user to the storage object stored in at least one of the NAS volume and the OSS bucket.
H04L 67/1097 - Protocoles dans lesquels une application est distribuée parmi les nœuds du réseau pour le stockage distribué de données dans des réseaux, p. ex. dispositions de transport pour le système de fichiers réseau [NFS], réseaux de stockage [SAN] ou stockage en réseau [NAS]
69.
Methods And Systems For Moving Virtual Logical Storage Units Between Data Center Management Servers
A method and system uses a first storage container for a first data center management server and a second storage container for a second data center management server that are associated with at least one common flexible logical storage unit in a storage system through a storage interface appliance to move a virtual logical storage unit from the first data center management server to the second data center management server. In order to move the virtual logical storage unit, the virtual logical storage unit is detached from a first virtual computing instance of the first data center management server and then attached to a second virtual computing instance of the second data center management server without any storage level modifications in the storage system.
G06F 9/455 - ÉmulationInterprétationSimulation de logiciel, p. ex. virtualisation ou émulation des moteurs d’exécution d’applications ou de systèmes d’exploitation
70.
ZERO-COPY VOLUME MOVE WITHIN A DISTRIBUTED STORAGE SYSTEM
Systems and methods for performing a zero-copy volume move between nodes of a distributed storage system are provided. In one example, an approach for performing a zero-copy volume move is proposed in which volume data may be maintained in place within a storage pod and need not be copied to move a given volume between the source node and the destination node. In one embodiment, file system metadata (e.g., a top-most physical volume block number (PVBN) of a node tree representing the volume at issue) of a write-anywhere file system is copied from the source node to the destination node. Since the storage pod is associated with a global PVBN space that is visible to all nodes of the distributed storage system, as a result of copying the top-mode PVBN of the volume to the destination node, anything below the top-most PVBN will automatically be visible to the destination node.
A data management system can include a disk unit and a set of controllers. The disk unit can contain, at least in part, a set of storage media, a first persistent memory, and a second persistent memory. The set of storage media can be configured to implement a storage space. The set of controllers can be configured to write to the storage space and to implement a set of nodes including a first node and a second node. The first node can be configured to generate and write first node journal data to the first persistent memory. The second node can be configured to obtain a failure indication for the first node, obtain the first node journal data from the second persistent memory, and generate and provide a reply to a backend using the first node journal data.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 11/07 - Réaction à l'apparition d'un défaut, p. ex. tolérance de certains défauts
72.
DATA FLOW BOTTLENECK MANAGEMENT TECHNIQUES USING CACHING MECHANISMS IN DISTRIBUTED STORAGE ENVIRONMENTS
Approaches to data flow bottleneck management using caching mechanisms in a distributed storage environment are disclosed. A request is received by a first data storage node having a first set of interface components, a first set of data management components, a first advisory cache, and a first set of data storage devices. The request has a corresponding file. The first advisory cache is checked for an entry corresponding to the file. The request is routed based on a file characteristic corresponding to the request if there is no corresponding entry in the first advisory cache or to a second data storage node based on the entry in the first advisory cache. Potential bottleneck conditions are monitored on the first node. An advisory cache entry in the first advisory cache is generated in response to determining that a bottleneck condition exists.
G06F 12/123 - Commande de remplacement utilisant des algorithmes de remplacement avec listes d’âge, p. ex. file d’attente, liste du type le plus récemment utilisé [MRU] ou liste du type le moins récemment utilisé [LRU]
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
G06F 12/0817 - Protocoles de cohérence de mémoire cache à l’aide de méthodes de répertoire
Techniques are provided for on-demand serverless disaster recovery. A primary node may host a primary volume. Snapshots of the primary volume may be backed up to an object store. In response to failure, a secondary node and/or an on-demand volume may be created on-demand. The secondary node may provide clients with failover access to the on-demand volume while a restore process restores a snapshot of the primary volume to the on-demand volume. In some embodiments, there was no secondary node and/or on-demand volume while the primary node was operational. This conserves computing resources that would be wasted by otherwise hosting the secondary node and/or on-demand volume while clients were able to access the primary volume through the primary node. Modifications directed to the on-demand volume are incrementally backed up to the object store for subsequently restoring the primary volume after recovery.
G06F 11/20 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel en utilisant un masquage actif du défaut, p. ex. en déconnectant les éléments défaillants ou en insérant des éléments de rechange
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
74.
METHODS AND STORAGE NODES TO DECREASE DELAY IN RESUMING INPUT OUTPUT (I/O) OPERATIONS AFTER A NON-DISRUPTIVE EVENT FOR A STORAGE OBECT OF A DISTRIBUTED STORAGE SYSTEM BY UTILIZING ASYNCHRONOUS INFLIGHT REPLAY OF THE I/O OPERATIONS
In one embodiment, a method comprises maintaining state information regarding a data replication status for a storage object of the storage node of a primary storage cluster with the storage object being replicated to a replicated storage object of a secondary storage cluster, temporarily disallowing input/output (I/O) operations when the storage object has a connection loss or failure. The method further includes initiating a resynchronization between the storage object and the replicated storage object including initiating asynchronous persistent inflight tracking and replay of any missing I/O operations that are missing from one of a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster, and allowing new I/O operations to be handled with the storage object of the primary storage cluster without waiting for completion of the asynchronous persistent inflight tracking and replay at the secondary storage cluster.
Techniques are provided for a cell based test automation framework. Conventional plain-text based test automation frameworks require extensive knowledge of scripting/programming languages to create and debug test scripts with numerous lines of code written in plain-text. Thus, conventional plain-text based test automation frameworks are disadvantageous in terms of time, effort, and training costs. The disclosed cell based test automation framework overcomes these disadvantages by utilizing spreadsheets defining a test script to execute. Each sheet within a spreadsheet defines a test case that the cell based test automation framework is to execute. The testing spreadsheet include cells storing test data and control values. The control values are used to control the flow of executing the test script to perform a test using the test data. The spreadsheet is very easy for a developer to create, and the cell based test automation framework outputs logs that are easy to read, understand, and debug.
Approaches to data flow bottleneck management using caching mechanisms in a distributed storage environment are disclosed. A read request is received by a first data storage node having a first set of interface module(s), a first set of data management module(s), a first redirection layer, and a first set of data storage devices. The read request has a corresponding file to be read. The first redirection layer is checked for an entry corresponding to the file. The read request is routed based on a file characteristic corresponding to the read request if there is no corresponding entry in the first redirection layer or to a second data storage node based on the entry in the first redirection layer. Potential bottleneck conditions are monitored on the first node. A redirection layer entry in the first redirection layer is generated in response to determining that a bottleneck condition exists.
The technology disclosed herein enables a higher-level process to perform storage volume management with knowledge of a physical storage backend underlying a storage volume. In a particular example, a method includes mounting a storage volume to a computing node of the computing nodes. The storage volume is stored in a storage pool of a plurality of underlying storage pools. The method further includes determining an identifier for the storage pool, receiving a request to duplicate the storage volume, and determining a second identifier for a second storage pool of the plurality of underlying storage pools to which the storage volume will be duplicated. When the second identifier matches the identifier, creating a clone of the storage volume rather than copying the storage volume to the second storage pool.
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
78.
Facilitating analytics involving cold data tiered to an object store by an on-premises storage solution by creating a virtual storage appliance on the object store
Systems and methods for creating a virtual storage appliance on an object store to which an on-premises storage solution has tiered cold data are provided. According to one embodiment, a physical storage system deployed in an on-premises environment replicates hot data of a file system of the physical storage system stored on a set of disks associated with the physical storage system to an object storage service of a hyperscaler to which cold data of the file system has previously been tiered out by the file system. An instance of a virtual storage system may then be created within the hyperscaler that utilizes the object storage service as a primary storage tier. Finally, analytics involving a dataset, including both the data and the cold data, may be performed by accessing the dataset via the virtual storage system.
The disclosed technology relates to managing input-output operation in a zoned storage system includes identifying a first physical zone and a second physical zone within a zoned namespace solid-state drive associated with a logical zone to perform a received write operation. Data to be written in the received write operation is temporarily staged in a zone random write area associated with the identified second physical zone. Based a storage threshold of the zone random write area, a determination is made regarding when to transfer temporarily staged data to be written area to the identified second physical zone. When the storage threshold of the zone random write area determined to have exceeded, temporarily staged data to be written is transferred to the identified second physical zone.
The disclosed technology relates to determining physical zone data within a zoned namespace solid state drive (SSD), associated with logical zone data included in a first received input-output operation based on a mapping data structure within a namespace of the zoned namespace SSD. A second input-output operation specific to the determined physical zone data is generated wherein the second input-output operation and the received input-output operation is of a same type. The generated second input-output operation is completed using the determined physical zone data within the zoned namespace SSD.
Disclosed are methods and systems for processing storage system monitoring data. The time component may be removed from the storage system monitoring data and may be stored as a mathematical formula, such as a line. The data from the storage system monitoring data may be separately analyzed to determine a compression algorithm to be applied to the data, such that the compressed data may be stored as a mathematical formula. Storing the storage system monitoring data as a mathematical formula lowers the amount of storage needed to store the storage system monitoring data, lowers the amount of network bandwidth needed to transfer the storage system monitoring data, and lowers the processing resources needed to process the storage system monitoring data. The removed time component may be used as an index to access the compressed data.
The technology disclosed herein enables worker nodes to reestablish connections when the connection protocols fail to successfully reconnect on their own. In a particular example, a method includes establishing a plurality of TCP-based connections (e.g., iSCSI and/or NVMe) between a plurality of worker nodes and a storage system and determining status of the plurality of TCP-based connections. When the status indicates a first connection of the plurality of TCP-based connections has failed, the method includes determining credentials used to establish the first connection are invalid. In response to determining the credentials are invalid, the method includes requesting new credentials from a controller node and reestablishing the first connection using the new credentials.
A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain compression information for a compressed segment generated from a segment of storage system monitoring data and generate a predicted portion of storage system monitoring data using the compression information. The monitoring system can obtain an additional portion of storage system monitoring data, the additional portion contiguous to the segment of storage system monitoring data, and determine that the predicted portion matches the additional portion and combining the additional portion and the compressed segment. In response to a user query, the monitoring system can perform at least one of: reconstructing and providing the additional portion using the compressed segment; or providing the compressed segment for reconstruction of the additional portion.
A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain a segment of storage system monitoring data and generate a compressed segment from the segment and a reconstructed segment from the compressed segment. The monitoring system can identify locations in the segment based on a comparison of the segment and the reconstructed segment and determine values for the identified locations. In response to a user query received from a user system, the user query indicating a portion of the segment, the monitoring system can perform at least one of: reconstructing and providing the portion using the identified locations, the determined values for the identified locations, and the compressed segment; or providing the identified locations, the determined values for the identified locations, and the compressed segment for reconstruction of the portion.
In one embodiment, a computer implemented method includes storing objects in a first bucket and storing files in a second bucket of a first storage cluster of the distributed storage system, synchronously replicating data of the objects into a third mirrored bucket of a second storage cluster of the distributed storage system, synchronously replicating OSP configuration data from the first storage cluster to the second storage cluster during the synchronous replication, and providing non-disruptive operations with zero recovery time objective (RTO), and ensuring consistency between the objects in the first bucket and the objects in the third bucket for a software application that is accessing one or more objects and files using the OSP. The objects and files are accessible through an object storage protocol (OSP).
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 16/27 - Réplication, distribution ou synchronisation de données entre bases de données ou dans un système de bases de données distribuéesArchitectures de systèmes de bases de données distribuées à cet effet
G06F 16/28 - Bases de données caractérisées par leurs modèles, p. ex. des modèles relationnels ou objet
Techniques are provided for migrating a volume utilizing backend object copy operations. Data of the volume is stored within objects stored across a storage tier and capacity tier of a source object store. As part of migrating the volume to a destination object store, the objects are migrated to the destination cluster. Directly copying the objects involves multiple read operations to the source object store and a write operation at the destination object store. The techniques provided herein improve the efficiency of the migration by initially sending metadata from the source object store to the destination object store for performing backend object copy operations to migrate the volume. This results in fewer operations and less network usage, thus improving the efficiency and cost of migrating the volume.
Techniques are provided for migrating a volume utilizing an object copy work queue and an object copy driver module. Data of the volume is stored within objects stored across a storage tier and capacity tier of a source object store. As part of migrating the volume to a destination object store, the objects are migrated to the destination cluster. Directly copying the objects involves multiple read operations to the source object store and a write operation at the destination object store. The techniques provided herein improve the efficiency of the migration by initially sending metadata from the source object store to the destination object store for performing backend object copy operations to migrate the volume. This results in fewer operations and less network usage, thus improving the efficiency and cost of migrating the volume.
The technology disclosed herein enables enforcement of Information Lifecycle Management (ILM) policies across tenants in an object storage system using tags associated with the ILM policies. In a particular example, a method includes identifying ILM policies for an object storage system having multiple tenants and associating respective tags with the ILM policies. The method further includes enabling a subset of the tags to be assigned to data objects of a tenant of the multiple tenants and enforcing a portion of the ILM policies on the data objects, wherein the portion of the ILM policies corresponds to tags of the subset assigned to the data objects.
A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can segment the obtained storage system monitoring data into storage system monitoring data segments. The monitoring system can then compress the storage system monitoring data segments into the compressed storage system monitoring data segments using at least two compression techniques. In response to a user query, the monitoring system can identify at least one compressed storage system monitoring data segment. The monitoring system can then perform at least one of reconstructing and providing a portion of the storage system monitoring data using the at least one compressed storage system monitoring data segment or providing the at least one compressed storage system monitoring data segment for reconstruction of the portion of the storage system monitoring data.
Disclosed herein are methods, systems, and apparatus for the detection of data integrity anomalies indicative of malware for a datastore of an organization. To identify an anomaly in a file, a portion of a file is identified to be used in a vector comparison. The portion can comprise sentences or paragraphs for text files, entries, rows, or columns for spreadsheet files, or some other divisible portion of a file. A vector having multiple dimensions is generated for the portion based on the content in the portion. Each dimension of the multiple dimensions corresponds to a feature of the portion. A variation is determined between the vector and one other vector associated with one other portion of the file. One or more actions to take with respect to the file is determined based on the variation, such as malware mitigation, and the action is performed with respect to the file.
A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain storage system monitoring data and generate a segment of the storage system monitoring data. The monitoring system can determine time information, a compression technique, and a compression parameter set for the segment. In response to the user query, the monitoring system can identify the compression technique and the compression parameter set based on a time interval of the query and the time information and perform at least one of: reconstructing and providing at least a portion of the segment using the time information, the compression technique, and the compression parameter set; or providing the compression parameter set and the indication of the compression technique for reconstruction of the portion.
A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain storage system monitoring data and generate a segment by applying the storage system monitoring data to a machine learning model trained to segment the storage system monitoring data. The monitoring system can generate a compressed segment by applying a specified compression technique to the segment. In response to user query, the user query specifying a portion of the storage system monitoring data; the monitoring system can perform at least one of: reconstructing and providing the portion using the compressed segment; or providing the compressed segment for reconstruction of the portion.
Disclosed herein are methods, systems, and apparatus for the detection of data integrity anomalies indicative of malware for a datastore of an organization. To identify an anomaly in a file, a portion of a file is identified to be used in a vector comparison. The portion can comprise sentences or paragraphs for text files, entries, rows, or columns for spreadsheet files, or some other divisible portion of a file. A vector having multiple dimensions is generated for the portion based on the content in the portion. Each dimension of the multiple dimensions corresponds to a feature of the portion. A variation is determined between the vector and one other vector associated with one other portion of the file. One or more actions to take with respect to the file is determined based on the variation, such as malware mitigation, and the action is performed with respect to the file
G06F 21/56 - Détection ou gestion de programmes malveillants, p. ex. dispositions anti-virus
G06F 16/583 - Recherche caractérisée par l’utilisation de métadonnées, p. ex. de métadonnées ne provenant pas du contenu ou de métadonnées générées manuellement utilisant des métadonnées provenant automatiquement du contenu
94.
Modified read path for processing read operations during object migration
Techniques are provided for modifying a read path to process read operations during a storage operation, such as an operation to migrate objects storing data of a volume. The objects are stored across a storage tier and capacity tier of a source object store. As part of migrating the volume to a destination object store, the objects are migrated to the destination cluster. Directly copying the objects involves multiple read operations to the source object store and a write operation at the destination object store. The techniques provided herein improve the efficiency of the migration by initially sending metadata from the source object store to the destination object store for performing backend block copy operations to migrate the volume.
G06F 16/185 - Systèmes de gestion de stockage hiérarchisé, p. ex. migration de fichiers ou politiques de migration de fichiers
G06F 11/10 - Détection ou correction d'erreur par introduction de redondance dans la représentation des données, p. ex. en utilisant des codes de contrôle en ajoutant des chiffres binaires ou des symboles particuliers aux données exprimées suivant un code, p. ex. contrôle de parité, exclusion des 9 ou des 11
95.
Methods to synchronously replicate data and manage audit configuration and audit data for a distributed storage system
In one embodiment, a computer implemented method includes comprises storing objects in a first bucket and files in a second bucket of a first storage cluster of the distributed storage system, initiating an audit job on the first storage cluster, synchronously replicating audit configuration data and mirroring audit data (e.g., audit files, logs) from the first storage cluster to the second storage cluster, performing a switchover process from the first storage cluster to the second storage cluster, and initiating an audit job on the second storage cluster based on the audit configuration during the switchover process. The first storage cluster initially handles input/output operations for a software application before the switchover process.
G06F 7/00 - Procédés ou dispositions pour le traitement de données en agissant sur l'ordre ou le contenu des données maniées
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 16/27 - Réplication, distribution ou synchronisation de données entre bases de données ou dans un système de bases de données distribuéesArchitectures de systèmes de bases de données distribuées à cet effet
G06F 16/28 - Bases de données caractérisées par leurs modèles, p. ex. des modèles relationnels ou objet
96.
AGGREGATE INLINE DEDUPLICATION WITH VOLUME GRANULAR ENCRYPTION
Techniques are provided for aggregate inline deduplication and volume granularity encryption. For example, data that is exclusive to a volume of a tenant is encrypted using an exclusive encryption key accessible to the tenant. The exclusive encryption key of that tenant is inaccessible to other tenants. Shared data that has been deduplicated and shared between the volume and another volume of a different tenant is encrypted using a shared encryption key of the volume. The shared encryption key is made available to other tenants. In this way, data can be deduplicated across multiple volumes of different tenants of a storage environment, while maintaining security and data privacy at a volume level.
The technology disclosed herein enhances data protection in a distributed storage system. In a particular example, a method includes determining a drive in a subject node of the distributed storage system has failed while storing first data of a data set distributed across nodes of the distributed storage system by a data protection mechanism. The method further includes broadcasting failure information indicating the data set from the subject node to other nodes of the distributed storage system. At the other nodes, in response to receiving the failure information, the method includes identifying a subset of the other nodes that also store a portion of the data set. In each identified node of the subset, the method includes identifying second data of the data set stored on a local drive and copying the second data to a different local drive to protect the data set from further drive failure.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
98.
Workload Analysis For Long-Term Management Via Performance Service Levels
Systems, methods, and machine-readable media for monitoring a storage system and assigning performance service levels to workloads running on nodes within a cluster are disclosed. A performance manager may estimate the performance demands of each workload within the cluster and assign a performance service level to each workload according to the performance requirements of the workload, and further taking into account an overall budgeting framework. The estimates are performed using historical performance data for each workload. A performance service level may include a service level object, a service level agreement, and latency parameters. These parameters may provide a ceiling to the number of operations per second that a workload may use without guaranteeing the use of the operations per second, a guaranteed number of operations per second that a workload may use before being throttled, and define the permitted delay in completing a request to the workload.
Disclosed are systems, computer-readable mediums, and methods for managing client performance in a storage system. According to one embodiment, a total Input/Output Operations per Second (IOPS) pool and a read/write IOPS pool are managed for clients to ensure their write requests can be accommodated by both pools. In one example, a write request is received from a client by the storage system. A requested number of write IOPS is determined for a time period to accommodate the request. Based on the requested number of write IOPS exceeding a number of allocated write IOPS to the client for the time period, a target total IOPS for the client during the time period is determined by subtracting the number of allocated write IOPS from a number of allocated total IOPS to the client. At least a portion of the request is performed by executing the target total IOPS during the time period.
Systems and methods for multiple device consumption of shared namespaces of ephemeral storage devices by a consumer of a virtual storage system are provided. In an example, multiple namespaces of respective ephemeral storage devices are shared among multiple consumers of a virtual storage system by creating multiple partitions within each of the namespaces for use by respective consumers of the multiple consumers. Corresponding partitions of respective shared namespace may then be treated as a stripe set to facilitate multiple device consumption for a subsystem (e.g., operation log journaling) of the virtual storage system by striping data associated with input/output (I/O) requests of a consumer (e.g., a journaling driver) across one or more stripe units of one or more stripes within the stripe set.
