README
Introduction
ChaosMeta is a cloud-native chaos engineering platform open sourced by Ant Group. It embodies the methodologies, technologies and products that Ant Group has accumulated over many years in the practice of large-scale red and blue offensive and defensive drills at the company level. With the "Risk Catalog" (internal general risk scenario manual for technical components in various fields) as theoretical guidance, combined with technical practice, it has escorted Ant Group's various promotional activities for many years.
ChaosMeta is a platform dedicated to supporting all stages of fault drills, covering platform capabilities in multiple stages such as access detection, traffic injection, fault injection, fault measurement, fault recovery, and recovery measurement. While liberating productivity for users, it is also pursuing the future form of chaos engineering: one-click automated drills, and even intelligent drills.
Core Advantages
Simple and easy to use, provides user interface, low threshold for use
Support visual user interface, Kubernetes API, command line, HTTP API, and other methods.
Fully verified by a large amount of practical experience, high reliability
The Blue Army team of Ant Group has been deeply involved in the chaos engineering industry for many years. It holds company-level large-scale red and blue offensive and defensive drills every year, facing all the company's businesses, and many businesses also conduct 7X24-hour drills and monthly normal drills
Internal drill object types cover cloud products, Kubernetes, Operator applications, databases (OceanBase, Etcd, etc.), middleware (message queues, distributed scheduling, configuration centers, etc.), business applications (Java applications, C++ applications, Golang applications)
High flexibility, supporting a variety of user needs
Whether the user wants a complete chaos engineering platform, or just wants the underlying platform capabilities such as remote injection, orchestration and scheduling, or even just wants the single-machine fault injection capability, or manages and injects targets on or off the cloud, there are corresponding deployment plans to meet
Rich fault injection capabilities, cloud native chaos engineering
Because Ant Group attaches great importance to offensive and defensive drills, it has led to large-scale and high-frequency drills, which in turn has promoted the construction of various fault injection capabilities. And because Ant has a huge internal infrastructure scale, coupled with the low fault tolerance of finance, the stability requirements for infrastructure such as Kubernetes and middleware are very high. Therefore, Ant Chaos Engineering has accumulated rich fault capabilities in the cloud-native field. and exercise experience.
The platform has powerful capabilities, supports the complete "chaos engineering life cycle", and is oriented towards automation.
ChaosMeta covers access detection, traffic injection, fault injection, fault measurement, fault recovery, recovery measurement and other stages of platform capabilities, as the technical basis of "automated chaos engineering"
In addition to the platform capability support of the exercise process, another big problem in the automated exercise is the design of the experiment. At present, it is difficult to completely rely on machines to automatically design. However, we can systematically abstract the reusable experience and organize it into a book. When conducting chaos engineering exercises on the same type of components, we can quickly reuse it. This is the original intention of the risk catalog design
ChaosMeta will realize the automated drill capability of one-click physical examination based on the technical foundation of "Chaos Engineering Life Cycle" and the theoretical basis of "Risk Catalog", directly generate the target stability score, and greatly liberate users in chaos
Architecture overview
Client layer (Client)
The Client layer is mainly composed of chaosmeta-platform components. Its main task is to lower the threshold for users to use and provide a visual interface to facilitate users' planning, orchestration, experiment configuration, experiment record details, and Agent management (pods/node of k8s clusters, cross-cluster objects, non-k8s physical machines/containers, etc.) and other platform capabilities.
Engine layer (Engine)
The Engine layer includes the core platform capabilities of ChaosMeta and the implementation of some cloud-native fault capabilities, including the following components:
chaosmeta-CRD: ChaosMeta's platform capabilities are developed based on the Operator framework, so each type of capability has a corresponding CRD, and then the corresponding Operator monitors the status and executes the corresponding operation. For example, the CRD of the fault injection capability is experiments.inject.chaosmeta.io and the operator corresponding to monitoring is chaosmeta-inject-operator. Therefore, users can create corresponding CR instances through Kubectl or Kubernetes-Client to perform corresponding functions;
chaosmeta-inject-operator: monitors the CR instance related to fault injection created by the user, compares the actual state of the CR in the cluster with the expected state in the control loop to execute the relevant fault injection logic and state flow, and tunes the actual state to the expected state. Perform different operations according to the fault type defined by the CR instance. For example, if it is a system resource fault, it needs to be injected remotely through chaosmeta-daemonset or HTTP or command channel; if it is a cloud-native fault, it will be injected based on Kubernetes APIServer , and if it involves a dynamic admission fault, it will also request chaosmeta-webhook to update the Mutating rules and Validating rules;
chaosmeta-webhook: The API processing flow of each APIServer needs to go through authentication, authentication, and admission, and the admission stage will go through the stages of Mutating Admission Webhook (tampering) and Validating Admission Webhook (verification), and chaosmeta-webhook will Define update resource matching rules, and perform operations such as interception, tampering, delay, and exceptions on the user's Kubernetes resource creation request. This is very meaningful for Operator applications and fault drill scenarios related to the robustness of Kubernetes' own clusters.
chaosmeta-measure-operator: This is the component used to perform the measurement function, mainly used in two phases: failure measurement and recovery measurement. The fault metric is an effective measure of the fault injection effect, and the recovery metric is the effective measure of the resilience of the defense platform. The ability to measure is the key ability to realize the automation and intelligence of chaos engineering.
For example, the failure effect of a drill is expected to be that the number of successful requests for a certain service drops by 50%, and the corresponding defense platform is expected to be discovered within 5 minutes and recovered within 10 minutes. The execution method is realized by fully using the CPU. Then in the fault measurement stage, it is necessary to find the time point (fault effective point) when the number of successful service requests drops by 50% compared with that before the fault injection. In the recovery measurement phase, it is necessary to find the time point when the corresponding alarm is generated (fault discovery point), and also to find the time point (fault recovery point) at which the success rate is requested to restore the water level before the drill after the fault discovery point. Finally, an analysis report of this drill is generated, and the areas for improvement of the defense platform are given.
chaosmeta-workflow-operator: Provides fault orchestration capabilities. Because in reality, except for a single failure scenario. There is also a need for a large number of complex fault scenarios, which require a series-parallel combination of different fault injection capabilities for simulation. And orchestration is not limited to fault injection, but also includes orchestration nodes with different capabilities such as traffic injection, fault access detection, fault measurement, recovery measurement, etc. This is also a key capability to automate the exercise.
chaosmeta-flow-operator: This is a component used to perform traffic injection, mainly used to mock the traffic of the target services. Because when we conduct fault drills, we often need to meet the flow rate to achieve the effect of the fault. For example, if you want to trigger a service delay alarm for a certain service, it is not enough to inject the delay into the container network of this service. If there is no traffic request, the corresponding monitoring alarm will not be triggered.
Kernel layer (Kernel)
The Kernel layer mainly includes the realization of single-machine fault injection capabilities, mainly including the chaosmetad component, which provides the way of resident HTTP service and command line execution, and also encapsulates the corresponding daemonset component (chaosmeta-daemonset). It can be flexibly matched with different training platforms.
Capabilities of the current version
The current version has released: user interface, fault injection scheduling engine, measurement engine, traffic injection engine, single machine fault injection tool and other components
User Interface
Provides experiment orchestration capabilities and lowers the threshold for use (the current version of the UI does not yet support traffic injection type and measurement type nodes);
Provides the ability to inject and filter remote targets of Pod/Node in the cluster (the UI will support targets outside the cluster in the future);
Provides space management capabilities and can separate and manage data on demand;
Provide account permission management system.
Fault injection capability
System Resources Exception: CPU, memory, network, disk, process, file, etc.;
Kernel Resource Exception: fd, nproc, etc.;
JVM Dynamic Injection: function call delay, function return value tampering, function throwing exception, etc.;
Container Fault Injection: kill container, suspend container, CPU, memory, network, disk, process, file, JVM injection and other experimental scenarios in the container;
Kubernetes Injection: execute experimental scenarios such as CPU, memory, network, disk, process, file, JVM injection on any pod;
Cloud-Native Faults: Abnormalities in cluster resources such as accumulation of a large number of Pending Pods and Completed Jobs; there are also abnormalities in instances of cloud-native resources such as Deployment, Node, and Pod, such as copy expansion and shrinking tampering of Deployment instances, and injection of Pod instance Finalizers.
Measuring Capabilities
monitor: Make expected judgments on the values of monitoring items, such as whether the CPU usage monitoring value of a certain machine is greater than 90%. Prometheus is supported by default.
pod: Make expected judgments on pod-related data, such as whether the number of pod instances of an application is greater than 3
http: Make expected judgments on http requests. For example, when making a specified http request, whether the return status code is 200
tcp: Make expected judgments on tcp requests, such as testing whether the 8080 port of a certain server is connectable
Traffic injection capability
http: http traffic injection
Getting Start
Quickly try the single-machine injection capability
Fault Ability Usage
For details, see: Function Instructions
Installation Guide
For details, see: Installation Guide
Comminicate
Welcome to submit defects, questions, suggestions and new features, all problems can be submitted to Github Issues, you can also contact us in the following ways:
DingTalk group: 21765030887
Slack group: ChaosMeta
WeChat public account: ChaosMeta混沌工程
Twitter：AntChaosMeta
Email: chaosmeta.io@gmail.com
WeChat group: email communication/WeChat public account to obtain QR code invitation
RoadMap
Platform capabilities
The future evolution of ChaosMeta platform capabilities is divided into three stages
Phase 1 - Manual Configuration
The goal to be achieved is to open all the components in the architecture diagram to the outside world. At this time, it can support the complete life cycle of chaos engineering, enter the field of primary automated chaos engineering, and use the "risk catalog" as a theoretical reference. Once manual configuration, multiple times automatically.
The order of opening to the outside world is as follows (if you have relevant needs, you are welcome to submit an issue, and priority adjustments will be considered):
Phase 2 - Automation
At this stage, the "Risk Catalog" will play a greater role. It not only gives the risk of a class of applications, but also the corresponding prevention and emergency recommendations, and the score of each item, and ChaosMeta will The "risk catalog" is integrated into a risk medical examination package of general components, which realizes the one-click "physical examination" capability, inputs target application information, and directly outputs a risk score and risk analysis report.
Phase 3 - intelligence
Explore the direction of combining artificial intelligence
Fault Injection Capability
The following is just a classification of fault capabilities. For the specific atomic fault capabilities provided, please refer to the description of fault capabilities (welcome to submit issues and put forward new capability requirements, and those with higher requirements are given priority):
License
ChaosMeta follows the Apache 2.0 license, please read LICENSE for details
Last updated
