MySQL

Percona XtraDB Cluster 5.7.17-27.20 is now available

MySQL Performance Blog - Fri, 2017-03-17 18:29

Percona announces the release of Percona XtraDB Cluster 5.7.17-27.20 on March 16, 2017. Binaries are available from the downloads section or our software repositories.

NOTE: You can also run Docker containers from the images in the Docker Hub repository.

Percona XtraDB Cluster 5.7.17-27.20 is now the current release, based on the following:

All Percona software is open-source and free. Details of this release can be found in the 5.7.17-27.20 milestone on Launchpad.

There are no new features or bug fixes to the main components, besides upstream changes and the following fixes related to packaging:

  • BLD-512: Fixed startup of garbd on Ubuntu 16.04.2 LTS (Xenial Xerus).
  • BLD-519: Added the garbd debug package to the repository.
  • BLD-569: Fixed grabd script to return non-zero if it fails to start.
  • BLD-570: Fixed service script for garbd on Ubuntu 16.04.2 LTS (Xenial Xerus) and Ubuntu 16.10 (Yakkety Yak).
  • BLD-593: Limited the use of rm and chown by mysqld_safe to avoid exploits of the CVE-2016-5617 vulnerability. For more information, see 1660265.
    Credit to Dawid Golunski (https://legalhackers.com).
  • BLD-610: Added version number to the dependency requirements of the full RPM package.
  • BLD-643: Fixed systemctl to mark mysql process as inactive after it fails to start and not attempt to start it again. For more information, see 1662292.
  • BLD-644: Added the which package to PXC dependencies on CentOS 7. For more information, see 1661398.
  • BLD-645: Fixed mysqld_safe to support options with a forward slash (/). For more information, see 1652838.
  • BLD-647: Fixed systemctl to show correct status for mysql on CentOS 7. For more information, see 1644382.

Help us improve our software quality by reporting any bugs you encounter using our bug tracking system. As always, thanks for your continued support of Percona!

Categories: MySQL

Column Store Database Benchmarks: MariaDB ColumnStore vs. Clickhouse vs. Apache Spark

MySQL Performance Blog - Fri, 2017-03-17 18:12

This blog shares some column store database benchmark results, and compares the query performance of MariaDB ColumnStore v. 1.0.7 (based on InfiniDB), Clickhouse and Apache Spark.

I’ve already written about ClickHouse (Column Store database).

The purpose of the benchmark is to see how these three solutions work on a single big server, with many CPU cores and large amounts of RAM. Both systems are massively parallel (MPP) database systems, so they should use many cores for SELECT queries.

For the benchmarks, I chose three datasets:

  1. Wikipedia page Counts, loaded full with the year 2008, ~26 billion rows
  2. Query analytics data from Percona Monitoring and Management
  3. Online shop orders

This blog post shares the results for the Wikipedia page counts (same queries as for the Clickhouse benchmark). In the following posts I will use other datasets to compare the performance.

Databases, Versions and Storage Engines Tested
  • MariaDB ColumnStore v. 1.0.7, ColumnStore storage engine
  • Yandex ClickHouse v. 1.1.54164, MergeTree storage engine
  • Apache Spark v. 2.1.0, Parquet files and ORC files

Although all of the above solutions can run in a “cluster” mode (with multiple nodes), I’ve only used one server.

Hardware

This time I’m using newer and faster hardware:

  • CPU: physical = 2, cores = 32, virtual = 64, hyperthreading = yes
  • RAM: 256Gb
  • Disk: Samsung SSD 960 PRO 1TB, NVMe card
Data Sizes

I’ve loaded the above data into Clickhouse, ColumnStore and MySQL (for MySQL the data included a primary key; Wikistat was not loaded to MySQL due to the size). MySQL tables are InnoDB with a primary key.

Dataset Size (GB) Column Store Clickhouse MySQL Spark / Parquet Spark / ORC file Wikistat 374.24 Gb 211.3 Gb n/a (> 2 Tb) 395 Gb 273 Gb Query metrics 61.23 Gb 28.35 Gb 520 Gb Store Orders 9.3 Gb 4.01 Gb 46.55 Gb

 

Query Performance

Wikipedia page counts queries

Test type (warm) Spark Clickhouse ColumnStore Query 1: count(*) 5.37 2.14 30.77 Query 2: group by month 205.75 16.36 259.09 Query 3: top 100 wiki pages by hits (group by path) 750.35 171.22 1640.7

Test type (cold) Spark Clickhouse ColumnStore Query 1: count(*) 21.93 8.01 139.01 Query 2: group by month 217.88 16.65 420.77 Query 3: top 100 wiki pages by hits (group by path) 887.434 182.56 1703.19


Partitioning and Primary Keys

All of the solutions have the ability to take advantage of data “partitioning,” and only scan needed rows.

Clickhouse has “primary keys” (for the MergeTree storage engine) and scans only the needed chunks of data (similar to partition “pruning” in MySQL). No changes to SQL or table definitions is needed when working with Clickhouse.

Clickhouse example:

:) select count(*), toMonth(date) as mon :-] from wikistat where toYear(date)=2008 :-] and toMonth(date) = 1 :-] group by mon :-] order by mon; SELECT count(*), toMonth(date) AS mon FROM wikistat WHERE (toYear(date) = 2008) AND (toMonth(date) = 1) GROUP BY mon ORDER BY mon ASC ┌────count()─┬─mon─┐ │ 2077594099 │ 1 │ └────────────┴─────┘ 1 rows in set. Elapsed: 0.787 sec. Processed 2.08 billion rows, 4.16 GB (2.64 billion rows/s., 5.28 GB/s.) :) select count(*), toMonth(date) as mon from wikistat where toYear(date)=2008 and toMonth(date) between 1 and 10 group by mon order by mon; SELECT count(*), toMonth(date) AS mon FROM wikistat WHERE (toYear(date) = 2008) AND ((toMonth(date) >= 1) AND (toMonth(date) <= 10)) GROUP BY mon ORDER BY mon ASC ┌────count()─┬─mon─┐ │ 2077594099 │ 1 │ │ 1969757069 │ 2 │ │ 2081371530 │ 3 │ │ 2156878512 │ 4 │ │ 2476890621 │ 5 │ │ 2526662896 │ 6 │ │ 2460873213 │ 7 │ │ 2480356358 │ 8 │ │ 2522746544 │ 9 │ │ 2614372352 │ 10 │ └────────────┴─────┘ 10 rows in set. Elapsed: 13.426 sec. Processed 23.37 billion rows, 46.74 GB (1.74 billion rows/s., 3.48 GB/s.)

As we can see here, ClickHouse has processed ~two billion rows for one month of data, and ~23 billion rows for ten months of data. Queries that only select one month of data are much faster.

For ColumnStore we need to re-write the SQL query and use “between ‘2008-01-01’ and 2008-01-10′” so it can take advantage of partition elimination (as long as the data is loaded in approximate time order). When using functions (i.e., year(dt) or month(dt)), the current implementation does not use this optimization. (This is similar to MySQL, in that if the WHERE clause has month(dt) or any other functions, MySQL can’t use an index on the dt field.)

ColumnStore example:

MariaDB [wikistat]> select count(*), month(date) as mon -> from wikistat where year(date)=2008 -> and month(date) = 1 -> group by mon -> order by mon; +------------+------+ | count(*) | mon | +------------+------+ | 2077594099 | 1 | +------------+------+ 1 row in set (2 min 12.34 sec) MariaDB [wikistat]> select count(*), month(date) as mon from wikistat where date between '2008-01-01' and '2008-01-31' group by mon order by mon; +------------+------+ | count(*) | mon | +------------+------+ | 2077594099 | 1 | +------------+------+ 1 row in set (12.46 sec)

Apache Spark does have partitioning however. It requires the use of partitioning with parquet format in the table definition. Without declaring partitions, even the modified query (“select count(*), month(date) as mon from wikistat where date between ‘2008-01-01’ and ‘2008-01-31’ group by mon order by mon”) will have to scan all the data.

The following table and graph shows the performance of the updated query:

Test type / updated query Spark Clickhouse ColumnStore group by month, one month, updated syntax 205.75 0.93 12.46 group by month, ten months, updated syntax 205.75 8.84 170.81

 

Working with Large Datasets

With 1Tb uncompressed data, doing a “GROUP BY” requires lots of memory to store the intermediate results (unlike MySQL, ColumnStore, Clickhouse and Apache Spark use hash tables to store groups by “buckets”). For example, this query requires a very large hash table:

SELECT path, count(*), sum(hits) AS sum_hits, round(sum(hits) / count(*), 2) AS hit_ratio FROM wikistat WHERE project = 'en' GROUP BY path ORDER BY sum_hits DESC LIMIT 100

As “path” is actually a URL (without the hostname), it takes a lot of memory to store the intermediate results (hash table) for GROUP BY.

MariaDB ColumnStore does not allow us to “spill” data on disk for now (only disk-based joins are implemented). If you need to GROUP BY on a large text field, you can decrease the disk block cache setting in Columnstore.xml (i.e., set disk cache to 10% of RAM) to make room for an intermediate GROUP BY:

<DBBC> <!-- The percentage of RAM to use for the disk block cache. Defaults to 86% --> <NumBlocksPct>10</NumBlocksPct>

In addition, as the query has an ORDER BY, we need to increase max_length_for_sort_data in MySQL:

ERROR 1815 (HY000): Internal error: IDB-2015: Sorting length exceeded. Session variable max_length_for_sort_data needs to be set higher. mysql> set global max_length_for_sort_data=8*1024*1024;

SQL Support SQL Spark* Clickhouse ColumnStore INSERT … VALUES ✅ yes ✅ yes ✅ yes INSERT SELECT / BULK INSERT ✅ yes ✅ yes ✅ yes UPDATE ❌ no ❌ no ✅ yes DELETE ❌ no ❌ no ✅ yes ALTER … ADD/DROP/MODIFY COLUMN ❌ no ✅ yes ✅ yes ALTER … change paritions ✅ yes ✅ yes ✅ yes SELECT with WINDOW functions ✅ yes ❌ no ✅ yes

 

*Spark does not support UPDATE/DELETE. However, Hive supports ACID transactions with UPDATE and DELETE statements. BEGIN, COMMIT, and ROLLBACK are not yet supported (only the ORC file format is supported).

ColumnStore is the only database out of the three that supports a full set of DML and DDL (almost all of the MySQL’s implementation of SQL is supported).

Comparing ColumnStore to Clickhouse and Apache Spark  Solution  Advantages  Disadvantages MariaDB ColumnStore
  • MySQL frontend (make it easy to migrate from MySQL)
  • UPDATE and DELETE are supported
  • Window functions support
  • Select queries are slower
  • No replication from normal MySQL server (planned for the future versions)
  • No support for GROUP BY on disk
Yandex ClickHouse
  • Fastest performance
  • Better compression
  • Primary keys
  • Disk-based GROUP BY, etc.
  • No MySQL protocol support
Apache Spark
  • Flexible storage options
  • Machine learning integration (i.e., pyspark ML libraries run inside spark nodes)
  • No MySQL protocol support
  • Slower select queries (compared to ClickHouse)

Conclusion

Yandex ClickHouse is an absolute winner in this benchmark: it shows both better performance (>10x) and better compression than
MariaDB ColumnStore and Apache Spark. If you are looking for the best performance and compression, ClickHouse looks very good.

At the same time, ColumnStore provides a MySQL endpoint (MySQL protocol and syntax), so it is a good option if you are migrating from MySQL. Right now, it can’t replicate directly from MySQL but if this option is available in the future we can attach a ColumnStore replication slave to any MySQL master and use the slave for reporting queries (i.e., BI or data science teams can use a ColumnStore database, which is updated very close to realtime).

Table Structure and List of Queries

Table structure (MySQL / Columnstore version):

CREATE TABLE `wikistat` ( `date` date DEFAULT NULL, `time` datetime DEFAULT NULL, `project` varchar(20) DEFAULT NULL, `subproject` varchar(2) DEFAULT NULL, `path` varchar(1024) DEFAULT NULL, `hits` bigint(20) DEFAULT NULL, `size` bigint(20) DEFAULT NULL ) ENGINE=Columnstore DEFAULT CHARSET=utf8

Query 1:

select count(*) from wikistat

Query 2a (full scan):

select count(*), month(dt) as mon from wikistat where year(dt)=2008 and month(dt) between 1 and 10 group by month(dt) order by month(dt)

Query 2b (for partitioning test)

select count(*), month(date) as mon from wikistat where date between '2008-01-01' and '2008-10-31' group by mon order by mon;

Query 3:

SELECT path, count(*), sum(hits) AS sum_hits, round(sum(hits) / count(*), 2) AS hit_ratio FROM wikistat WHERE project = 'en' GROUP BY path ORDER BY sum_hits DESC LIMIT 100;

 

Categories: MySQL

Percona Server for MongoDB: Dashing New LDAP Authentication Plugin

MySQL Performance Blog - Thu, 2017-03-16 22:06

This blog post is another in the series on the Percona Server for MongoDB 3.4 bundle release. In this blog, we’ll look at the new LDAP authentication plugin. 

Hear ye, hear ye, hear ye… With the arrival of version 3.4, Percona has included an LDAP plugin in Percona Server for MongoDB. Authentication is an essential part in client communication with the database backend.

What is LDAP?

LDAP stands for Lightweight Directory Access Protocol. It’s a centralized environment containing information on users or services. This information can be objects like passwords, roles or other meta information. Typically when connecting to the MongoDB server, you simply have to authenticate with the MongoDB servers local credential lists. Using an external authentication method like the one included in Percona Server for MongoDB, you can poll an external service. External authentication allows the MongoDB server to verify the client’s username and password against a separate service, such as OpenLDAP or Active Directory.

Why should you use it?

Having a centralized LDAP offers you the ability to rely on one single “truth” for authentication and authorization. LDAP is essential in large organizations where maintaining users on a big infrastructure becomes troublesome.

In ideal situations, you would use your LDAP authentication for multiple MongoDB servers, and even other database solutions like MySQL. The idea is that you only need to modify the passwords or accounts centrally, so you can manage entries without having to modify them locally on each MongoDB instance.

Having a centralized authentication repository is often a requirement when managing highly sensitive information (due to compliancy requirements). A central repository for user information, like an LDAP server, solves the problem of a central source for authentication. When a user with database level privileges leaves the organization, simply shutting off the one central LDAP account will prevent access to all databases that use it as a source. If local accounts were created without being tied back to a central user repository, then the likelihood of an access revocation getting missed is far greater. This is why many security standards require accounts to be created with an LDAP/Active Directory type of service.

So what components do we actually use?

If you  want to visualize it in a figure:

  • SASL Daemon. Used as a MongoDB server-local proxy for the remote LDAP service. This service is used for MongoDB as an intermediate service. It will translate the request and poll the LDAP server.
  • SASL Library. Used by the MongoDB client and server to create data necessary for the authentication mechanism. This library is used by the MongoDB client and server for making properly formatted requests to the SASL daemon.
So how does it work?

An authentication session uses the following sequence:

  1. A MongoDB client connects to a running mongod instance.
  2. The client creates a PLAIN authentication request using the SASL library.
  3. The client then sends this SASL request to the server as a special MongoDB command.
  4. The mongod server receives this SASL Message, with its authentication request payload.
  5. The server then creates a SASL session scoped to this client, using its own reference to the SASL library.
  6. Then the server passes the authentication payload to the SASL library, which in turn passes it on to the saslauthd daemon.
  7. The saslauthd daemon passes the payload on to the LDAP service to get a YES or NO authentication response (in other words, does this user exist and is the password correct).
  8. The YES/NO response moves back from saslauthd, through the SASL library, to mongod.
  9. The mongod server uses this YES/NO response to authenticate the client or reject the request.
  10. If successful, the client has authenticated and can proceed.

Below a  visualisation of the authentication path using the LDAP connector

An example of the output when using the authentication plugin

The mongod server is running with the added option:

cat /etc/default/mongod OPTIONS="-f /etc/mongod.conf --auth --setParameter authenticationMechanisms=PLAIN,SCRAM-SHA-1" STDOUT="/var/log/mongodb/mongod.stdout" STDERR="/var/log/mongodb/mongod.stderr" First let’s make a user in MongoDB, I’ve created an Organisational Unit and associated user with password on LDAP > db.getSiblingDB("$external").createUser({ ... user : 'dim0', ... roles: [ {role : "read", db: 'percona'} ] ... }) Successfully added user: { "user" : "utest", "roles" : [ { "role" : "read", "db" : "percona" } ] }

At this point the user is correctly added on MongoDB.

Let’s try and perform a read query on the database “percona”:

db.getSiblingDB("$external").auth({mechanism: "PLAIN", user: 'dim0', pwd: 'test’, digestPassword: false }) 1 use percona db.foo.find() { "_id" : ObjectId("58b3e4b80322deccc99dc763"), "x" : 1 } { "_id" : ObjectId("58b3e8fee48bdc7edeb31cc5"), "x" : 2 } { "_id" : ObjectId("58b3e931e48bdc7edeb31cc6"), "x" : 3 } > exit

Now let’s try and write something in it:

db.foo.insert({x : 5}) WriteResult({ "writeError" : { "code" : 13, "errmsg" : "not authorized on percona to execute command { insert: "foo", documents: [ { _id: ObjectId('58b3e97f2343c5da97a2256e'), x: 5.0 } ], ordered: true }" } })

This is logical behavior, as we only allowed read interaction on the percona database.

After a correct login, you will find the following in the mongod.log:

2017-02-27T08:55:19.612+0000 I ACCESS   [conn2] Successfully authenticated as principal dim0 on $external

If an incorrect login happens, the following entry will appear in the mongod.log:

2017-02-27T09:10:55.297+0000 I ACCESS   [conn4] PLAIN authentication failed for dim0 on $external from client 127.0.0.1:34812 ; OperationFailed: SASL step did not complete: (authentication failure)

Conclusion

Percona Server for MongoDB has an easy way of integrating correctly with SASL authd. If you are looking for an option of centrally managing the users of your MongoDB environments, look no further. Keep in mind, however, that if you don’t need a centrally managed environment adding this functionality creates additional complexity to your infrastructure. You can find additional information on the LDAP plugin in our documentation at https://www.percona.com/doc/percona-server-for-mongodb/ext_authentication/index.html.

Categories: MySQL

Monitoring Databases: A Product Comparison

MySQL Performance Blog - Thu, 2017-03-16 20:12

In this blog post, I will discuss the solutions for monitoring databases (which includes alerting) I have worked with and recommended in the past to my clients. This survey will mostly focus on MySQL solutions. 

One of the most common issues I come across when working with clients is monitoring and alerting. Many times, companies will fall into one of these categories:

  • No monitoring or alerting. This means they have no idea what’s going on in their environment whatsoever.
  • Inadequate monitoring. Maybe people in this camp are using a platform that just tells them the database is up or connections are happening, but there is no insight into what the database is doing.
  • Too much monitoring and alerting. Companies in this camp have tons of dashboards filled with graphs, and their inbox is full of alerts that get promptly ignored. This type of monitoring is just as useful as the first option. Alert fatigue is a real thing!

With my clients, I like to talk about what monitoring they need and what will work for them.

Before we get started, I do want to point out that I have borrowed some text and/or graphics from the websites and promotional material of some of the products I’m discussing.

Simple Alerting

Percona provides a Nagios plugin for database alerts: https://www.percona.com/downloads/percona-monitoring-plugins/.

I also like to point out to clients what metrics are important to monitor long term to make sure there are no performance issues. I prefer the following approach:

  • On the hardware level:
    • Monitor CPU, IO, network usage and how it trends monthly. If some resource consumption comes to a critical level, this might be a signal that you need more capacity.
  • On the MySQL server level:
    • Monitor connections, active threads, table locks, row locks, InnoDB IO and buffer pool usage
    • For replication, monitor seconds behind master (SBM), binlog size and replication errors. In Percona XtraDB Cluster, you might want to watch wsrep_local_recv_queue.
  • On the query level:
    • Regularly check query execution and response time, and make sure it stays within acceptable levels. When execution time approaches or exceeds established levels, evaluate ways to optimize your queries.
  • On the application side:
    • Monitor that response time is within established SLAs.
High-Level Monitoring Solution Comparison PMM MonYOG Severalnines VividCortex SelectStar Databases Supported MySQL, MongoDB and others with custom addons MySQL MySQL, MongoDB, PostgreSQL MySQL, MongoDB, PostgreSQL, Redis MySQL, MongoDB, PostgreSQL, Hadoop, Cassandra, Amazon Dynamo, IBM DB2, SQL Server, Oracle Open Source x Cost Free Subscription per node Subscription per node and free Community Edition Subscription per instance Subscription per instance Cloud or
On Premises
On premises On premises On premises Cloud with on premises collector Cloud with on premises collector Has Agents x x Monitoring x x x x x Alerting Yes, but requires custom setup x x x x Replication Topology Management x x Query Analytics x x x x Configuration Management x x Backup Management x OS Metrics x x  x x x Configuration Advisors x  x x Failover Management x x ProxySQL and
HA Proxy Support
Monitors ProxySQL x

 

PMM

http://pmmdemo.percona.com

https://www.percona.com/blog/2016/04/18/percona-monitoring-and-management/

https://www.percona.com/doc/percona-monitoring-and-management/index.html

Percona Monitoring and Management (PMM) is a fully open source solution for managing MySQL platform performance and tuning query performance. It allows DBAs and application developers to optimize the performance of the database layer. PMM is an on-premises solution that keeps all of your performance and query data inside the confines of your environment, with no requirement for data to cross the Internet.

Assembled from a supported package of “best-of-breed” open source tools such as Prometheus, Grafana and Percona’s Query Analytics, PMM delivers results right out of the box.

With PMM, anyone with database maintenance responsibilities can get more visibility for actionable enhancements, realize faster issue resolution times, increase performance through focused optimization and better manage resources. More information allows you to concentrate efforts on the areas that yield the highest value, rather than hunting and pecking for speed.

PMM monitors and provides performance data for Oracle’s MySQL Community and Enterprise Servers, as well as Percona Server for MySQL and MariaDB.

Alerting

In the current version of PMM, custom alerting can be set up. Percona has a guide here: https://www.percona.com/blog/2017/01/23/mysql-and-mongodb-alerting-with-pmm-and-grafana/.

Architecture

The PMM platform is based on a simple client-server model that enables efficient scalability. It includes the following modules:

  • PMM Client is installed on every MySQL host that you want to monitor. It collects MySQL server metrics, general system metrics, and query analytics data for a complete performance overview. Collected data is sent to the PMM Server.
  • PMM Server aggregates collected data and presents it in the form of tables, dashboards and graphs in a web interface.

MySQL Configuration

Percona recommends certain settings to get the most out of PMM. You can get more information and a guide here: https://www.percona.com/doc/percona-monitoring-and-management/conf-mysql.html.

Advantages
  • Fast setup
  • Fully supported and backed by Percona
  • Impressive roadmap ahead
  • Monitors your database in depth
  • Query analytics
  • Quick setup docker container
  • Free and open source
Disadvantages
  • New, could still have some growing pains
  • Requires agents on database machines
Severalnines

http://severalnines.com/

Severalnines ClusterControl provides access to 100+ key database and host metrics that matter to your operational performance. You can visualize historical performance in custom dashboards to establish operational baselines and capacity planning. It lets you proactively monitor and receive advice to address immediate and potential database and server issues, and ships with over 100 built-in advisors or easily-writeable custom advisors for your specific needs. It is very scriptable and customizable with some effort.

Severalnines has a free community version as well as a commercial offering. The free version includes deployment, monitoring and advisors with a Developer Studio (with which users can create their own advisors).

Severalnines is definitely more sysadmin focused. The best part about it is its ability to deploy and manage deployments of your database with almost no command line work.

The community edition of ClusterControl is “free forever”.

Architecture

ClusterControl is an agentless management and automation software for database clusters. It helps deploy, monitor, manage and scale your database server/cluster directly from ClusterControl user interface.

ClusterControl consists of four components:

Component Package Naming Role ClusterControl controller (cmon) clustercontrol- controller The brain of ClusterControl. A backend service performing automation, management, monitoring and scheduling tasks. All the collected data will be stored directly inside CMON database ClusterControl REST API clustercontrol-cmonapi Interprets request and response data between ClusterControl UI and CMON database ClusterControl UI clustercontrol A modern web user interface to visualize and manage the cluster. It interacts with CMON controller via remote procedure call (RPC) or REST API interface ClusterControl NodeJS clustercontrol-nodejs This optional package is introduced in ClusterControl version 1.2.12 to provide an interface for notification services and integration with 3rd party tools

 

Advantages
  • Agentless
  • Monitors, deploys and manages:
    • Database
    • Configuration
    • Backups
    • Users
  • Simple web GUI to manage your databases, alerts, users, settings
  • Can create custom monitors or jobs
  • Can off-load and compress backups
  • Great support team
  • Rich feature set and multiple databases supported
Disadvantages
  • Cost per node
  • UI can occasionally be clunky
  • Query tools lack as compared to other solutions here
  • Metrics and Advisors may not be as powerful or easy to use as other products
MONyog

https://www.webyog.com/product/monyog

MONyog MySQL Monitor and Advisor is a “MySQL DBA in a box” that helps MySQL DBAs manage more MySQL servers, tune their current MySQL servers and find and fix problems with their MySQL database applications before they can become serious problems or costly outages.

MONyog proactively monitors enterprise database environments and provides expert advice on how even those new to MySQL can tighten security, optimize performance and reduce downtime of their MySQL powered systems.

MONyog is more DBA focused and focuses on the MySQL configuration and queries.

Architecture

MONyog web server runs on Linux, monitoring MySQL on all platforms and also monitoring OS-data on Linux servers. To retrieve OS metrics, MONyog uses SSH. However, with this scenario (MONyog installed on a Linux machine) MONyog web-server/agent cannot collect Windows OS metrics.

Of course, the client where the MONyog output is viewed can be any browser supporting AJAX on any platform. MONyog can be installed on a remote PC as well as the server. It does not require processing, and with agentless monitoring it can collect and retrieve data from the server.

Advantages
  • Setup and startup within two minutes
  • Agentless
  • Good query tools
  • Manages configuration
  • Great advisors for database tuning built-in
  • Most comprehensive and detailed alerting
Disadvantages
  • Cost per node
  • Only supports MySQL
VividCortex

VividCortex is a good cloud-based tool to see what your production databases are doing. It is a modern SaaS database performance monitoring platform that significantly eases the pain of database performance at scale, on distributed and polyglot systems, for the entire engineering team. It’s hosted for you with industry-leading security, and is continuously improved and maintained. VividCortex measures and analyzes the system’s work and resource consumption. The result is an immediate insight into query performance, better performance and quality, faster time-to-market and reduced cost and effort.

Architecture

VividCortex is the combination of agent programs, APIs and a web application. You install the agents on your servers, they send data to their APIs, and you access the results through the web application at https://app.vividcortex.com. VividCortex has a diagram on their site showing how it works:

The agents are self-supervising, managed by an agent called vc-agent-007. You can read more about the agents in the agent-specific documentation. They send primarily time-series metrics to the APIs, at one-second granularity. It sometimes sends additional metadata as well. For example, query digests are required to show what query is responsible for specific query-related metrics.
On the backend, a distributed, fully multi-tenant service stores your data separately from all other customers. VividCortex servers are currently hosted in the Amazon AWS public cloud.

Advantages
  • Great visibility into query-level performance to pinpoint optimization efforts
  • Granularity, with the ability to identify performance fluctuations down to a one-second resolution
  • Smart anomaly detection using advanced statistics and machine learning to reduce false-positives and make alerts meaningful and actionable
  • Unique collaboration tools, enabling developers to answer many of their own questions and freeing DBAs to be more responsive and proactive.
Disadvantages
  • Cloud-based tools may not be desirable in a secure environment
  • Cost
  • Not useful if you lose outside network access during an incident
  • Dependent on AWS availability
SelectStar

https://selectstar.io

SelectStar monitors key metrics for many different database types, and has a comprehensive alerts and recommendations system. SelectStar supports monitoring and alerts on:

  • MySQL, Percona Server for MySQL, MariaDB
  • PostgreSQL
  • Oracle
  • MongoDB
  • Microsoft SQL
  • DB2
  • Amazon RDS and Aurora
  • Hadoop
  • Cassandra

The alerts and recommendations are designed to ensure you have an immediate understanding of key issues — and where they are coming from. You can pinpoint the exact database instance that may be causing the issue, or go further up the chain and see if it’s an issue impacting several database instances at the host level.

Recommendations are often tied to alerts — if you have a red alert, there’s going to be a recommendation tied to it on how you can improve. However, the recommendations pop up even if your database is completely healthy — ensuring that you have visibility into how you can improve your configuration before you actually have an issue impacting performance.

Architecture

Using agentless collectors, SelectStar gathers data from both your on-premises and AWS platforms so that you can have insight into all of your database instances.

The collector is an independent machine within your infrastructure that pulls data from your database. It is low impact in order to not impact performance. This is a different approach from all of the other monitoring tools I have looked at.

Advantages
  • Multiple database technologies (the most out of the tools presented here)
  • Great visibility into query-level performance to pinpoint optimization efforts
  • Agentless
  • Good query tools
  • Great advisors for database tuning built in
  • Good alerting
  • Fast setup
  • Monitors your database in depth
  • Query analytics
Disadvantages
  • Cloud-based tools may not be desirable in a secure environment
  • Cost
  • New, could still have some growing pains
  • Still requires an on-premises collector
So What Do I Recommend?

It depends.” – Peter Z., CEO Percona

As always, I recommend whatever works best for your workload, in your environment, and within the standards of your company’s practices!

Categories: MySQL

Percona Live Featured Session with Evan Elias: Automatic MySQL Schema Management with Skeema

MySQL Performance Blog - Wed, 2017-03-15 22:03

Welcome to another post in the series of Percona Live featured session blogs! In these blogs, we’ll highlight some of the session speakers that will be at this year’s Percona Live conference. We’ll also discuss how these sessions can help you improve your database environment. Make sure to read to the end to get a special Percona Live 2017 registration bonus!

In this Percona Live featured session, we’ll meet Evan Elias, Director of Engineering, Tumblr. His session is Automatic MySQL Schema Management with SkeemaSkeema is a new open source CLI tool for managing MySQL schemas and migrations. It allows you to easily track your schemas in a repository, supporting a pull-request-based workflow for schema change submission, review, and execution.

I had a chance to speak with Evan about Skeema:

Percona: How did you get into database technology? What do you love about it?

Evan: I first started using MySQL at a college IT job in 2003, and over the years I eventually began tackling much larger-scale deployments at Tumblr and Facebook. I’ve spent most of the past decade working on social networks, where massive high-volume database technology is fundamental to the product. I love the technical challenges present in that type of environment, as well as the huge potential impact of database automation and tooling. In companies with giant databases and many engineers, a well-designed automation system can provide a truly enormous increase in productivity.

Percona: Your talk is called Automatic MySQL Schema Management with Skeema. What is Skeema, and how is it helpful for engineers and DBAs?

Evan: Skeema is an open source tool for managing MySQL schemas and migrations. It allows users to diff, push or pull schema definitions between the local filesystem and one or more databases. It can be configured to support multiple environments (e.g. development/staging/production), external online schema change tools, sharding, and service discovery. Once configured, an engineer or DBA can use Skeema to execute an online schema change on many shards concurrently simply by editing a CREATE TABLE statement in a file and then running “skeema push”.

Percona: What are the benefits of storing schemas in a repository?

Evan: The whole industry is moving towards infrastructure-as-code solutions, providing automated configuration which is reproducible across multiple environments. In extending this concept to database schemas, a file repository stores the desired state of each table, and a schema change is tied to simply changing these files. A few large companies like Facebook have internal closed-source tools to tie MySQL schemas to a git repo, allowing schema changes to be powered by pull requests (without any manual DBA effort). There hasn’t previously been an open source, general-purpose tool for managing schemas and migrations in this way, however. I developed Skeema to fill this gap.

Percona: What do you want attendees to take away from your session? Why should they attend?

Evan: In this session, MySQL DBAs will learn how to automate their schema change workflow to reduce manual operational work, while software engineers will discover how Skeema permits easy online migrations even in frameworks like Rails or Django. Skeema is a brand new tool, and this is the first conference session to introduce it. At this relatively early stage, feedback and feature requests from attendees will greatly influence the direction and prioritization of future development.

Percona: What are you most looking forward to at Percona Live 2017?

Evan: Percona Live is my favorite technical conference. It’s the best place to learn about all of the recent developments in the database world, and meet the top experts in the field. This is my fifth year attending in Santa Clara. I’m looking forward to reconnecting with old friends and making some new ones as well!

Register for Percona Live Data Performance Conference 2017, and see Evan present his session on Automatic MySQL Schema Management with Skeema. Use the code FeaturedTalk and receive $100 off the current registration price!

Percona Live Data Performance Conference 2017 is the premier open source event for the data performance ecosystem. It is the place to be for the open source community as well as businesses that thrive in the MySQL, NoSQL, cloud, big data and Internet of Things (IoT) marketplaces. Attendees include DBAs, sysadmins, developers, architects, CTOs, CEOs, and vendors from around the world.

The Percona Live Data Performance Conference will be April 24-27, 2017 at the Hyatt Regency Santa Clara & The Santa Clara Convention Center.

Categories: MySQL

Upgrading to Percona Server for MongoDB 3.4 from Previous Versions

MySQL Performance Blog - Wed, 2017-03-15 20:58

This post is another in the series on the Percona Server for MongoDB 3.4 bundle release. In this blog post, we will walk through upgrading to Percona Server for MongoDB 3.4 from a previous MongoDB version. We will highlight the important details to guarantee the successful completion of this task in your environment.

MongoDB 3.4 was just released, and it has a lot of new features such as views, a better sharding algorithm, facet aggregation, numeric precision data type and more.

The procedure below covers upgrading from a previous version. The process for a fresh installation is slightly different. We don’t need to enable compatibility (as explained later).

Before upgrading, please be aware of a few details/requirements:

  • Minimum version. The minimum version required to upgrade to MongoDB 3.4 is 3.2.8.
  • Config servers. Config servers only work as replica sets. The previous method – instances as config servers – is no longer supported.
  • Driver. The driver must be ready for the new features (such as views and decimal data type). Please make sure that your driver version contains all the required settings.
How to upgrade a MongoDB config server instance (the previous config server architecture) to a replica set
  1. Check if the balancer is running. If it is, disable the balancer process:

$ mongos mongos> sh.isBalancerRunning() false mongos> sh.stopBalancer() Waiting for active hosts... Waiting for the balancer lock... Waiting again for active hosts after balancer is off... WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 0 })

  1. Connect to the first MongoDB config server and run the command below. Please replace the _id and members with your own settings:

configsvr> rs.initiate( { _id: "configRS", configsvr: true, version: 1, members: [ { _id: 0, host: "tests:27019" } ] }) { "ok" : 1 }

Get the storage engine name that is used by this instance. We will use this information later. If you are using MMAPv1, there are some additional commands to run:

db.serverStatus().storageEngine { "name" : "wiredTiger", "supportsCommittedReads" : true, "persistent" : true }

Use the following command to read all the application startup parameters (informational only):

db.runCommand('getCmdLineOpts').parsed { "net" : { "port" : 27019 }, "processManagement" : { "fork" : true }, "sharding" : { "clusterRole" : "configsvr" }, "storage" : { "dbPath" : "config1" }, "systemLog" : { "destination" : "file", "path" : "config1/logconfig1.log" } }

  1. Restart the first MongoDB config instance with the –replSet configRS and –configsvrMode=sccc parameters:

mongod --configsvr --replSet configRS --configsvrMode=sccc --storageEngine wiredTiger --port 27019 --dbpath config1 --logpath &nbsp;config1/logconfig1.log --fork

If using a configuration file, you must use the following parameters:

sharding: clusterRole: configsvr configsvrMode: sccc replication: replSetName: csReplSet net: port: <port> storage: dbPath: <path> engine: <storageEngine>

  1. After the first config server has been restarted, start two empty config servers with the parameters settings chosen before. If you are using MMAPv1, please start three instances instead of two.

./mongod --configsvr --replSet configRS --port 27025 --dbpath rsconfig1 --logpath rsconfig1/rsconfig1 --fork ./mongod --configsvr --replSet configRS --port 27026 --dbpath rsconfig2 --logpath rsconfig2/rsconfig2 --fork (only if using MMAPv1) ./mongod --configsvr --replSet configRS --port 27027 --dbpath rsconfig2 --logpath rsconfig3/rsconfig3 --fork

  1. Connect to the first config server again and add the new instances to the replica set:

configRS:PRIMARY> rs.add({host : "tests:27025", priority : 0 , votes : 0}) { "ok" : 1 } configRS:PRIMARY> rs.add({host : "tests:27026", priority : 0 , votes : 0}) { "ok" : 1 } (only if using MMAPv1) rs.add({host : "tests:27027", priority : 0 , votes : 0}) { "ok" : 1 }

  1. Check the replication status. All the new instances must be in secondary status (part of the output has been omitted in the rs.status (below):

rs.status() { "set" : "configRS", "date" : ISODate("2017-02-07T13:11:12.914Z"), "myState" : 1, "term" : NumberLong(1), "configsvr" : true, "heartbeatIntervalMillis" : NumberLong(2000), "members" : [ { "_id" : 0, "name" : "tests:27019", "stateStr" : "PRIMARY", (...) }, { "_id" : 1, "name" : "tests:27025", "stateStr" : "SECONDARY", (...) }, { "_id" : 2, "name" : "tests:27026", "stateStr" : "SECONDARY", (...) }, { "_id" : 3, // (will appear only if using MMAP) "name" : "tests:27027", "stateStr" : "SECONDARY", (...) }, ], "ok" : 1 }

  1. Once the replica set is up and running, please stop old config instances running without replica set. Also, add the votes and priority to all members of the replica set. If using MMAPv1, remove votes and priority from cfg.members[0];:

var cfg = rs.conf(); cfg.members[0].priority = 1; // 0 if using MMAP cfg.members[1].priority = 1; cfg.members[2].priority = 1; cfg.members[0].votes = 1; // 0 if using MMAP cfg.members[1].votes = 1; cfg.members[2].votes = 1; (Only if the first config server is using mmap) cfg.members[3].priority = 1; cfg.members[3].votes = 1; rs.reconfig(cfg);

  1. Restart the mongos service using the new replica set repl/hostnames:port and the —configdb parameter:

./mongos --configdb configRS/tests:27019,tests:27025,tests:27026 --logpath mongos.log --fork

  1. Perform a stepDown() and shutDown() in the first server. If this server is using MMAPv1, remove it from the replica set using rs.remove(). If it is using wiredTiger, restart without configsvrMode=sccc in the parameters.

At this point, you have an instance that has been correctly migrated to a replica set.

  1. Please enable the balancer again (use a 3.4 client, or it will fail):

sh.startBalancer()

How to upgrade config instances server to version 3.4

Please follow these instructions to upgrade a shard to the 3.4 version.

As we already have the configs correctly running as replica sets, we need to upgrade the binaries versions. It is easy to do. Stop and replace the secondaries binaries versions:

  1. Connect to the mongos and stop the balancer.
  2. Stop the secondary, replace the old binary with the new version, and start the service.
  3. When the primary is the only one running with version 3.2, run rs.stepDown()  and force this instance to become secondary.
  4. Replace these (now) secondary binaries with the 3.4 version.

For the shards, add a new parameter before restarting the process in the 3.4 version:

  1. Stop the secondaries, replace the binaries and start it with the –shardsvr parameter.

If using a config file, add (this is a new parameter in this version):

sharding: clusterRole: shardsvr

  1. Step down the primary and perform this same process on it.
  2. After all the shards binaries have been changed to the 3.4 version, upgrade the mongos binaries.
  3. Stop mongos and replace this executable with the 3.4 version.

At this point, all the shards are running the 3.4 version, but by default the new features in MonogDB 3.4 are disabled. If we try to use a decimal data type, mongos reports the following:

WriteResult({ "nInserted" : 0, "writeError" : { "code" : 22, "errmsg" : "Cannot use decimal BSON type when the featureCompatibilityVersion is 3.2. See http://dochub.mongodb.org/core/3.4-feature-compatibility." } })

In order to enable MongoDB 3.4 features we need run the following in the mongos:

db.adminCommand( { setFeatureCompatibilityVersion: "3.4" } ) { "ok" : 1 } mongos> use testing switched to db testing mongos> db.foo.insert({x : NumberDecimal(10.2)}) WriteResult({ "nInserted" : 1 })

After changing the featureCompatibility to “3.4,”  all the new MongoDB 3.4 features will be available.

I hope this tutorial was useful in explaining upgrading to Percona Server for MongoDB 3.4. Please ping on twitter @adamotonete or @percona for any questions and suggestions.

Categories: MySQL

How to Restore a Single InnoDB Table from a Full Backup After Accidentally Dropping It

MySQL Performance Blog - Wed, 2017-03-15 20:31

In this blog post, we’ll look at how to restore a single InnoDB table from a full backup after dropping the table.

You can also see an earlier blog post about restoring a single table from a full backup here: How to recover a single InnoDB table from a full backup.

The idea behind the actions in that blog is based on the “Transportable Tablespace” concept, which was introduced in MySQL 5.6. So when you have deleted the data from a table, you are going to quickly restore this table as follows:

  • Prepare the backup
  • Discard the tablespace of the original table
  • Copy .ibd from the backup to the original table path
  • Import the tablespace

Of course, you need to test it using the process in production, even though it is relatively straightforward.

But how about when you drop a table? It is still a great process because you will lose the table structure and the datafiles.

The actions mentioned in the previous blog will not work here, simple because it is impossible to discard a non-existing tablespace.

Categories: MySQL

Webinar Thursday, March 16, 2017: Moving to Amazon Web Services (AWS)

MySQL Performance Blog - Mon, 2017-03-13 19:36

Join Percona’s Solutions Engineer Dimitri Vanoverbeke on Thursday, March 16, 2017 at 7:00 a.m. PDT / 10:00 a.m. EDT (UTC-7) for a webinar on Moving to Amazon Web Services (AWS).

Register Now This webinar covers the many challenges faced when migrating applications from on-premises into Amazon Web Services (AWS). It will specifically look at moving MySQL to AWS’s Relations Database Service (RDS) platform. AWS is a great platform for hosting your infrastructure in the cloud. This webinar will go over the particulars of moving towards the DBaaS solution inside Amazon’s web services, and covers the different levels of service and options available.

The webinar will also discuss RDS specifics and possible migration techniques for pushing your information into RDS:

  • Moving to an RDS instance specifics
  • Optimizing configuration on RDS
  • Choosing between EC2 with MySQL and RDS
  • Moving to Amazon Aurora
  • Selecting between availability options in Amazon RDS
  • Using backups with Amazon RDS

This webinar provides a good overview on what migrating to RDS can bring to your organization, and many of the essential configuration options.

Register for the webinar here.

Dimitri Vanoverbeke, Solutions Engineer

At the age of 7, Dimitri received his first computer. Since then, he has been addicted to anything with a digital pulse. Dimitri has been active in IT professionally since 2003. He took various roles from internal system engineering to consulting.

Prior to joining Percona, Dimitri worked as an open source consultant for a leading open source software consulting firm in Belgium. During his career, Dimitri became familiar with a broad range of open source solutions and with the devops philosophy. Whenever he’s not glued to his computer screen, he enjoys traveling, cultural activities, basketball and the great outdoors. Dimitri is living with his girlfriend in the beautiful city of Ghent, Belgium.

Categories: MySQL

Percona XtraDB Cluster 5.6.35-26.20 is now available

MySQL Performance Blog - Mon, 2017-03-13 17:32

Percona announces the release of Percona XtraDB Cluster 5.6.35-26.20 on March 10, 2017. Binaries are available from the downloads section or our software repositories.

Percona XtraDB Cluster 5.6.35-26.20 is now the current release, based on the following:

All Percona software is open-source and free. Details of this release can be found in the 5.6.35-26.20 milestone on Launchpad.

There are no new features or bug fixes to the main components, besides upstream changes and the following fixes related to packaging:

  • BLD-593: Limited the use of rm and chown by mysqld_safe to avoid exploits of the CVE-2016-5617 vulnerability. For more information, see 1660265.
    Credit to Dawid Golunski (https://legalhackers.com).
  • BLD-610: Added version number to the dependency requirements of the full RPM package.
  • BLD-645: Fixed mysqld_safe to support options with a forward slash (/). For more information, see 1652838.

Help us improve our software quality by reporting any bugs you encounter using our bug tracking system. As always, thanks for your continued support of Percona!

Categories: MySQL

Troubleshooting MySQL access privileges issues: Q & A

MySQL Performance Blog - Sat, 2017-03-11 00:36

In this blog, I will provide answers to the Q & A for the Troubleshooting MySQL Access Privileges Issues webinar.

First, I want to thank everybody for attending the February 23 webinar. The recording and slides for the webinar are available here. Below is the list of your questions that I wasn’t able to answer during the webinar, with responses:

Q: Should the root@localhost user be given ALL privileges or Super privileges? Does All include Super privileges also?

A: Yes, you should have a user with all privileges. Better if this user has access from localhost only. ALL  includes SUPER.

Q: We have users who connect via a laptop that get dynamic IP addresses, so granting access with a server name is an easier way to manage these users. Can I grant access to a MySQL database with a hostname as opposed to an ipaddress? For example “myname@mymachine.mydomain.com” as opposed to “myname@10.10.10.10”?  Is the host cache/performance_schema required for this?

A: Yes, you can.

But it looks like I was not clear about host cache. Host cache is an internal structure that is always available and contains answers from DNS server. You cannot enable or disable it. Until version 5.6, you also could not control it. For example, if the cache got corrupted the only thing you could do is to restart the server. Version 5.6 the table HOST_CACHE was introduced to Performance Schema. With this table you can examine the content of the host cache and truncate it if needed.

Q: If there are multiple entries in the user table that match the connecting user (e.g., with wildcards, hostname, and IP), what rules does MySQL use to select which is used for authentication?  Does it try multiple ones until it gets a password match?

A: Not, mysqld does not try to hack your passwords. Instead it sorts the user table by name and host in descending order as I showed on slide #37 (page 110). Then it takes the first matching row. So if you created users foo@somehost, foo@some% and foo@1.2.3.4, and you connect as foo from somehost, mysqld first checks the user name and then chooses the first matching row foo@somehost. If you instead connect as foo from someotherhost, mysqld chooses foo@some%. An IP-based host is chosen if either mysqld started with option skip-networking or if 1.2.3.4  points to a host whose name does not start with “some”.

Mixing IP-based and name-based hosts is dangerous in situations when the same host can be resolved as somehost or 1.2.3.4. In this case, if something goes wrong with the host cache or DNS server, the wrong entry from the user table can be chosen. For example, if you initially had three hosts: uniquehost (which resolves as 1.2.3.4), somehost (which resolves as 4.3.2.1) and someothershost (which resolves as 4.3.2.2). Now you decided to re-locate uniquehost to a machine with IP 1.2.3.5 and use IP 1.2.3.4 for the host with name someyetanotherhost. In this case, the clients from the machine with IP 1.2.3.4 will be treated as foo@some%, which isn’t what you want.

To demonstrate this issue, I created two users and granted two different privileges to them:

mysql> create user sveta@Thinkie; Query OK, 0 rows affected (0,01 sec) mysql> create user sveta@'192.168.0.4'; Query OK, 0 rows affected (0,00 sec) mysql> grant all on *.* to 'sveta'@'Thinkie'; Query OK, 0 rows affected (0,00 sec) mysql> grant all on db1.* to 'sveta'@'192.168.0.4'; Query OK, 0 rows affected (0,00 sec)

Now I modified my /etc/hosts  file and pointed address 192.168.0.4  to name Thinkie:

127.0.0.1 localhost # 127.0.1.1 Thinkie 192.168.0.4 Thinkie

Now, if I connect as sveta both Thinkie and 192.168.0.4 are resolved to the same host:

sveta@Thinkie:$ mysql -hThinkie -usveta ... mysql> select user(), current_user(); +---------------+----------------+ | user() | current_user() | +---------------+----------------+ | sveta@Thinkie | sveta@thinkie | +---------------+----------------+ 1 row in set (0,00 sec) mysql> show grants; +--------------------------------------------------+ | Grants for sveta@thinkie | +--------------------------------------------------+ | GRANT ALL PRIVILEGES ON *.* TO 'sveta'@'thinkie' | +--------------------------------------------------+ 1 row in set (0,00 sec) mysql> q Bye sveta@Thinkie:$ mysql -h192.168.0.4 -usveta ... mysql> show grants; +--------------------------------------------------+ | Grants for sveta@thinkie | +--------------------------------------------------+ | GRANT ALL PRIVILEGES ON *.* TO 'sveta'@'thinkie' | +--------------------------------------------------+ 1 row in set (0,00 sec) mysql> select user(), current_user(); +---------------+----------------+ | user() | current_user() | +---------------+----------------+ | sveta@Thinkie | sveta@thinkie | +---------------+----------------+ 1 row in set (0,00 sec) mysql> q Bye

Now I modified the /etc/hosts  file and pointed Thinkie  back to 127.0.0.1  (localhost):

127.0.0.1 localhost 127.0.1.1 Thinkie # 192.168.0.4 Thinkie

But host 192.168.0.4 still resolves to Thinkie:

sveta@Thinkie:$ mysql -h192.168.0.4 -usveta ... mysql> select user(), current_user(); +---------------+----------------+ | user() | current_user() | +---------------+----------------+ | sveta@Thinkie | sveta@thinkie | +---------------+----------------+ 1 row in set (0,00 sec) mysql> show grants; +--------------------------------------------------+ | Grants for sveta@thinkie | +--------------------------------------------------+ | GRANT ALL PRIVILEGES ON *.* TO 'sveta'@'thinkie' | +--------------------------------------------------+ 1 row in set (0,00 sec) mysql> q Bye

The reason for this is a stalled host cache, which can be easily observable with Performance Schema:

sveta@Thinkie:$ mysql -uroot ... mysql> select * from performance_schema.host_cacheG *************************** 1. row *************************** IP: 192.168.0.4 HOST: Thinkie HOST_VALIDATED: YES SUM_CONNECT_ERRORS: 0 COUNT_HOST_BLOCKED_ERRORS: 0 COUNT_NAMEINFO_TRANSIENT_ERRORS: 0 COUNT_NAMEINFO_PERMANENT_ERRORS: 0 COUNT_FORMAT_ERRORS: 0 COUNT_ADDRINFO_TRANSIENT_ERRORS: 0 COUNT_ADDRINFO_PERMANENT_ERRORS: 0 COUNT_FCRDNS_ERRORS: 0 COUNT_HOST_ACL_ERRORS: 0 COUNT_NO_AUTH_PLUGIN_ERRORS: 0 COUNT_AUTH_PLUGIN_ERRORS: 0 COUNT_HANDSHAKE_ERRORS: 0 COUNT_PROXY_USER_ERRORS: 0 COUNT_PROXY_USER_ACL_ERRORS: 0 COUNT_AUTHENTICATION_ERRORS: 0 COUNT_SSL_ERRORS: 0 COUNT_MAX_USER_CONNECTIONS_ERRORS: 0 COUNT_MAX_USER_CONNECTIONS_PER_HOUR_ERRORS: 0 COUNT_DEFAULT_DATABASE_ERRORS: 0 COUNT_INIT_CONNECT_ERRORS: 0 COUNT_LOCAL_ERRORS: 0 COUNT_UNKNOWN_ERRORS: 0 FIRST_SEEN: 2017-03-02 23:19:32 LAST_SEEN: 2017-03-02 23:20:31 FIRST_ERROR_SEEN: NULL LAST_ERROR_SEEN: NULL 1 row in set (0,00 sec) mysql> truncate performance_schema.host_cache; Query OK, 0 rows affected (0,00 sec) mysql> q Bye

After I truncated table host_cache the numeric host resolves as I expect:

sveta@Thinkie:$ mysql -h192.168.0.4 -usveta ... mysql> show grants; +----------------------------------------------------------+ | Grants for sveta@192.168.0.4 | +----------------------------------------------------------+ | GRANT USAGE ON *.* TO 'sveta'@'192.168.0.4' | | GRANT ALL PRIVILEGES ON `db1`.* TO 'sveta'@'192.168.0.4' | +----------------------------------------------------------+ 2 rows in set (0,00 sec) mysql> select user(), current_user(); +-------------------+-------------------+ | user() | current_user() | +-------------------+-------------------+ | sveta@192.168.0.4 | sveta@192.168.0.4 | +-------------------+-------------------+ 1 row in set (0,00 sec) mysql> q Bye

Q: What privileges are required for a non-root or non-super user to be to use mysqldump to dump the database and then restore it on a different server?

A: Generally you should have SELECT  privilege on all objects you are going to dump. If you are dumping views, you also should have SHOW VIEW  privilege in order to run SHOW CREATE TABLE. If you want to dump stored routines/events, you need access to them as well. If you use option --lock-tables or --lock-all-tables, you should have the LOCK  privilege.

Q: If the max_connection value is reached in MySQL, can root@localhost with ALL privilege still login, or with Super privilege user can login?

A: ALL includes SUPER, so a user with ALL  privilege can login. Just note there can be only one such connection, thus do not grant SUPER  or ALL privilege to the application user.

Q: Is it possible to remove a priv at a lower level? In other words, grant select and delete at the database level, but remove delete for a specific table?  Or can privs only be added to?

A: Not, MySQL will reject such a statement:

mysql> show grants for sveta@'192.168.0.4'; +----------------------------------------------------------+ | Grants for sveta@192.168.0.4 | +----------------------------------------------------------+ | GRANT USAGE ON *.* TO 'sveta'@'192.168.0.4' | | GRANT ALL PRIVILEGES ON `db1`.* TO 'sveta'@'192.168.0.4' | +----------------------------------------------------------+ 2 rows in set (0,00 sec) mysql> revoke update on db1.t1 from sveta@'192.168.0.4'; ERROR 1147 (42000): There is no such grant defined for user 'sveta' on host '192.168.0.4' on table 't1'

Q: How can we have DB user roles… like a group of grants for a particular role?

A: You have several options.

  1. Use MariaDB 10.0.5 or newer. You can read about roles support in MariaDB here
  2. Use MySQL 8.0. You can read about roles in MySQL 8.0 here
  3. With MySQL 5.7: imitate roles as I showed on slide 19 (pages 53 – 60)
  4. With MySQL 5.5 and 5.6: use the same method as shown on slides, but use the custom authentication plugin that supports proxy users.
  5. Always: create a template with privileges, assign privileges to each user manually.

Q: How would you migrate role simulation with proxy users to actual roles in MySQL 8.x?

A: I would drop the proxied user and create a role with the same privileges instead, then grant the proxy user the newly created role instead of PROXY.

Q: Is there a plugin to integrate Active Directory and MySQL in order to use Active Directory groups?

A: There is commercial Windows Authentication Plugin, available in versions 5.5 and newer. You can also use the open source Percona PAM authentication plugin and connect it to Active Directory the same way as can be done for LDAP. There is a blog post describing how to do so, but I’ve never used this method myself.

Q: Can we use central auth with MySQL?

A: Yes, with the help of the PAM Plugin. There are tutorials for LDAP and Active Directory. You may use similar methods to setup other kinds of authentications, such as Kerberos.

Categories: MySQL

Percona Monitoring and Management (PMM) Graphs Explained: Custom MongoDB Graphs and Metrics

MySQL Performance Blog - Sat, 2017-03-11 00:10

This blog post is another in the series on the Percona Server for MongoDB 3.4 bundle release. In this blog post, we will cover how to add custom MongoDB graphs to Percona Monitoring and Management (PMM) and (for the daring Golang developers out there) how to add custom metrics to PMM’s percona/mongodb_exporter metric exporter.

To get to adding new graphs and metrics, we first need to go over how PMM gets metrics from your database nodes and how they become graphs.

Percona Monitoring and Management (PMM)

Percona Monitoring and Management (PMM) is an open-source platform for managing and monitoring MySQL and MongoDB. It was developed by Percona on top of open-source technology. Behind the scenes, the graphing features this article covers use Prometheus (a popular time-series data store), Grafana (a popular visualisation tool), mongodb_exporter (our MongoDB database metric exporter) plus other technologies to provide database and operating system metric graphs for your database instances.

Prometheus

As mentioned, Percona Monitoring and Management uses Prometheus to gather and store database and operating system metrics. Prometheus works on an HTTP(s) pull-based architecture, where Prometheus “pulls” metrics from “exporters” on a schedule.

To provide a detailed view of your database hosts, you must enable two PMM monitoring services for MongoDB graphing capabilities:

  1. linux:metrics
  2. mongodb:metrics

See this link for more details on adding monitoring services to Percona Monitoring and Management: https://www.percona.com/doc/percona-monitoring-and-management/pmm-admin.html#adding-monitoring-services

It is important to note that not all metrics gathered by Percona Monitoring and Management are graphed. This is by design. Storing more metrics vs. what is graphed is very useful when more advanced insight is necessary. We also aim for PMM to be simple and straightforward to use, explaining why we don’t graph all of the nearly 1,000 metrics we collect per MongoDB node on each polling.

My personal monitoring philosophy is “monitor until it hurts and then take one step back.” In other words, try to get a much data as you can without impacting the database or adding monitoring resources/cost. Also, Prometheus stores large volumes of metrics efficiently due to compression and highly optimized data structures on disk. This offsets a lot of the cost of collecting extra metrics.

Later in this blog, I will show how to add a graph for an example metric that is currently gathered by PMM but is not graphed in PMM as of today. To see what metrics are available on PMM’s Prometheus instance, visit “http://<pmm-server>/promtheus/graph”.

prometheus/node_exporter (linux:metrics)

PMM’s OS-level metrics are provided to Prometheus via the 3rd-party exporter: prometheus/node_exporter.

This exporter provides 712 metrics per “pull” on my test CentOS 7 host that has:

  • 2 x CPUs
  • 3 x disks with 6 x LVMs
  • 2 x network interfaces

Note: more physical or virtual devices add to the number of node_exporter metrics.

The inner workings and addition of metrics to this exporter will not be covered in this blog post. Generally, the current metrics offered by node_exporter are more than enough.

Below is a full example of a single “pull” of metrics from node_exporter on my test host:

$ curl -sk https://192.168.99.10:42000/metrics|grep node # HELP node_boot_time Node boot time, in unixtime. # TYPE node_boot_time gauge node_boot_time 1.488904144e+09 # HELP node_context_switches Total number of context switches. # TYPE node_context_switches counter node_context_switches 1.0407839e+07 # HELP node_cpu Seconds the cpus spent in each mode. # TYPE node_cpu counter node_cpu{cpu="cpu0",mode="guest"} 0 node_cpu{cpu="cpu0",mode="idle"} 6437.71 node_cpu{cpu="cpu0",mode="iowait"} 24 node_cpu{cpu="cpu0",mode="irq"} 0 node_cpu{cpu="cpu0",mode="nice"} 0 node_cpu{cpu="cpu0",mode="softirq"} 1.38 node_cpu{cpu="cpu0",mode="steal"} 0 node_cpu{cpu="cpu0",mode="system"} 117.65 node_cpu{cpu="cpu0",mode="user"} 33.6 node_cpu{cpu="cpu1",mode="guest"} 0 node_cpu{cpu="cpu1",mode="idle"} 6443.68 node_cpu{cpu="cpu1",mode="iowait"} 15.03 node_cpu{cpu="cpu1",mode="irq"} 0 node_cpu{cpu="cpu1",mode="nice"} 0 node_cpu{cpu="cpu1",mode="softirq"} 7.15 node_cpu{cpu="cpu1",mode="steal"} 0 node_cpu{cpu="cpu1",mode="system"} 118.08 node_cpu{cpu="cpu1",mode="user"} 28.84 # HELP node_disk_bytes_read The total number of bytes read successfully. # TYPE node_disk_bytes_read counter node_disk_bytes_read{device="dm-0"} 2.09581056e+08 node_disk_bytes_read{device="dm-1"} 1.114112e+06 node_disk_bytes_read{device="dm-2"} 5.500416e+06 node_disk_bytes_read{device="dm-3"} 2.98752e+06 node_disk_bytes_read{device="dm-4"} 3.480576e+06 node_disk_bytes_read{device="dm-5"} 4.0983552e+07 node_disk_bytes_read{device="dm-6"} 303104 node_disk_bytes_read{device="sda"} 2.40276992e+08 node_disk_bytes_read{device="sdb"} 4.5549568e+07 node_disk_bytes_read{device="sdc"} 8.967168e+06 node_disk_bytes_read{device="sr0"} 49152 # HELP node_disk_bytes_written The total number of bytes written successfully. # TYPE node_disk_bytes_written counter node_disk_bytes_written{device="dm-0"} 4.1160192e+07 node_disk_bytes_written{device="dm-1"} 0 node_disk_bytes_written{device="dm-2"} 2.413568e+06 node_disk_bytes_written{device="dm-3"} 9.0589184e+07 node_disk_bytes_written{device="dm-4"} 4.631552e+06 node_disk_bytes_written{device="dm-5"} 2.140672e+06 node_disk_bytes_written{device="dm-6"} 0 node_disk_bytes_written{device="sda"} 4.3257344e+07 node_disk_bytes_written{device="sdb"} 6.772224e+06 node_disk_bytes_written{device="sdc"} 9.3002752e+07 node_disk_bytes_written{device="sr0"} 0 # HELP node_disk_io_now The number of I/Os currently in progress. # TYPE node_disk_io_now gauge node_disk_io_now{device="dm-0"} 0 node_disk_io_now{device="dm-1"} 0 node_disk_io_now{device="dm-2"} 0 node_disk_io_now{device="dm-3"} 0 node_disk_io_now{device="dm-4"} 0 node_disk_io_now{device="dm-5"} 0 node_disk_io_now{device="dm-6"} 0 node_disk_io_now{device="sda"} 0 node_disk_io_now{device="sdb"} 0 node_disk_io_now{device="sdc"} 0 node_disk_io_now{device="sr0"} 0 # HELP node_disk_io_time_ms Milliseconds spent doing I/Os. # TYPE node_disk_io_time_ms counter node_disk_io_time_ms{device="dm-0"} 6443 node_disk_io_time_ms{device="dm-1"} 41 node_disk_io_time_ms{device="dm-2"} 319 node_disk_io_time_ms{device="dm-3"} 61024 node_disk_io_time_ms{device="dm-4"} 1159 node_disk_io_time_ms{device="dm-5"} 772 node_disk_io_time_ms{device="dm-6"} 1 node_disk_io_time_ms{device="sda"} 6965 node_disk_io_time_ms{device="sdb"} 2004 node_disk_io_time_ms{device="sdc"} 60718 node_disk_io_time_ms{device="sr0"} 5 # HELP node_disk_io_time_weighted The weighted # of milliseconds spent doing I/Os. See https://www.kernel.org/doc/Documentation/iostats.txt. # TYPE node_disk_io_time_weighted counter node_disk_io_time_weighted{device="dm-0"} 9972 node_disk_io_time_weighted{device="dm-1"} 46 node_disk_io_time_weighted{device="dm-2"} 369 node_disk_io_time_weighted{device="dm-3"} 147704 node_disk_io_time_weighted{device="dm-4"} 1618 node_disk_io_time_weighted{device="dm-5"} 968 node_disk_io_time_weighted{device="dm-6"} 1 node_disk_io_time_weighted{device="sda"} 10365 node_disk_io_time_weighted{device="sdb"} 3213 node_disk_io_time_weighted{device="sdc"} 143822 node_disk_io_time_weighted{device="sr0"} 5 # HELP node_disk_read_time_ms The total number of milliseconds spent by all reads. # TYPE node_disk_read_time_ms counter node_disk_read_time_ms{device="dm-0"} 7037 node_disk_read_time_ms{device="dm-1"} 46 node_disk_read_time_ms{device="dm-2"} 312 node_disk_read_time_ms{device="dm-3"} 56 node_disk_read_time_ms{device="dm-4"} 115 node_disk_read_time_ms{device="dm-5"} 906 node_disk_read_time_ms{device="dm-6"} 1 node_disk_read_time_ms{device="sda"} 7646 node_disk_read_time_ms{device="sdb"} 1192 node_disk_read_time_ms{device="sdc"} 412 node_disk_read_time_ms{device="sr0"} 5 # HELP node_disk_reads_completed The total number of reads completed successfully. # TYPE node_disk_reads_completed counter node_disk_reads_completed{device="dm-0"} 11521 node_disk_reads_completed{device="dm-1"} 133 node_disk_reads_completed{device="dm-2"} 1139 node_disk_reads_completed{device="dm-3"} 130 node_disk_reads_completed{device="dm-4"} 199 node_disk_reads_completed{device="dm-5"} 2077 node_disk_reads_completed{device="dm-6"} 42 node_disk_reads_completed{device="sda"} 13429 node_disk_reads_completed{device="sdb"} 2483 node_disk_reads_completed{device="sdc"} 1376 node_disk_reads_completed{device="sr0"} 12 # HELP node_disk_reads_merged The number of reads merged. See https://www.kernel.org/doc/Documentation/iostats.txt. # TYPE node_disk_reads_merged counter node_disk_reads_merged{device="dm-0"} 0 node_disk_reads_merged{device="dm-1"} 0 node_disk_reads_merged{device="dm-2"} 0 node_disk_reads_merged{device="dm-3"} 0 node_disk_reads_merged{device="dm-4"} 0 node_disk_reads_merged{device="dm-5"} 0 node_disk_reads_merged{device="dm-6"} 0 node_disk_reads_merged{device="sda"} 12 node_disk_reads_merged{device="sdb"} 0 node_disk_reads_merged{device="sdc"} 0 node_disk_reads_merged{device="sr0"} 0 # HELP node_disk_sectors_read The total number of sectors read successfully. # TYPE node_disk_sectors_read counter node_disk_sectors_read{device="dm-0"} 409338 node_disk_sectors_read{device="dm-1"} 2176 node_disk_sectors_read{device="dm-2"} 10743 node_disk_sectors_read{device="dm-3"} 5835 node_disk_sectors_read{device="dm-4"} 6798 node_disk_sectors_read{device="dm-5"} 80046 node_disk_sectors_read{device="dm-6"} 592 node_disk_sectors_read{device="sda"} 469291 node_disk_sectors_read{device="sdb"} 88964 node_disk_sectors_read{device="sdc"} 17514 node_disk_sectors_read{device="sr0"} 96 # HELP node_disk_sectors_written The total number of sectors written successfully. # TYPE node_disk_sectors_written counter node_disk_sectors_written{device="dm-0"} 80391 node_disk_sectors_written{device="dm-1"} 0 node_disk_sectors_written{device="dm-2"} 4714 node_disk_sectors_written{device="dm-3"} 176932 node_disk_sectors_written{device="dm-4"} 9046 node_disk_sectors_written{device="dm-5"} 4181 node_disk_sectors_written{device="dm-6"} 0 node_disk_sectors_written{device="sda"} 84487 node_disk_sectors_written{device="sdb"} 13227 node_disk_sectors_written{device="sdc"} 181646 node_disk_sectors_written{device="sr0"} 0 # HELP node_disk_write_time_ms This is the total number of milliseconds spent by all writes. # TYPE node_disk_write_time_ms counter node_disk_write_time_ms{device="dm-0"} 2935 node_disk_write_time_ms{device="dm-1"} 0 node_disk_write_time_ms{device="dm-2"} 57 node_disk_write_time_ms{device="dm-3"} 146868 node_disk_write_time_ms{device="dm-4"} 1503 node_disk_write_time_ms{device="dm-5"} 62 node_disk_write_time_ms{device="dm-6"} 0 node_disk_write_time_ms{device="sda"} 2797 node_disk_write_time_ms{device="sdb"} 2096 node_disk_write_time_ms{device="sdc"} 143569 node_disk_write_time_ms{device="sr0"} 0 # HELP node_disk_writes_completed The total number of writes completed successfully. # TYPE node_disk_writes_completed counter node_disk_writes_completed{device="dm-0"} 2701 node_disk_writes_completed{device="dm-1"} 0 node_disk_writes_completed{device="dm-2"} 70 node_disk_writes_completed{device="dm-3"} 415913 node_disk_writes_completed{device="dm-4"} 2737 node_disk_writes_completed{device="dm-5"} 17 node_disk_writes_completed{device="dm-6"} 0 node_disk_writes_completed{device="sda"} 3733 node_disk_writes_completed{device="sdb"} 3189 node_disk_writes_completed{device="sdc"} 421341 node_disk_writes_completed{device="sr0"} 0 # HELP node_disk_writes_merged The number of writes merged. See https://www.kernel.org/doc/Documentation/iostats.txt. # TYPE node_disk_writes_merged counter node_disk_writes_merged{device="dm-0"} 0 node_disk_writes_merged{device="dm-1"} 0 node_disk_writes_merged{device="dm-2"} 0 node_disk_writes_merged{device="dm-3"} 0 node_disk_writes_merged{device="dm-4"} 0 node_disk_writes_merged{device="dm-5"} 0 node_disk_writes_merged{device="dm-6"} 0 node_disk_writes_merged{device="sda"} 147 node_disk_writes_merged{device="sdb"} 19 node_disk_writes_merged{device="sdc"} 2182 node_disk_writes_merged{device="sr0"} 0 # HELP node_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which node_exporter was built. # TYPE node_exporter_build_info gauge node_exporter_build_info{branch="master",goversion="go1.7.4",revision="2d78e22000779d63c714011e4fb30c65623b9c77",version="1.1.1"} 1 # HELP node_exporter_scrape_duration_seconds node_exporter: Duration of a scrape job. # TYPE node_exporter_scrape_duration_seconds summary node_exporter_scrape_duration_seconds{collector="diskstats",result="success",quantile="0.5"} 0.00018768700000000002 node_exporter_scrape_duration_seconds{collector="diskstats",result="success",quantile="0.9"} 0.000199063 node_exporter_scrape_duration_seconds{collector="diskstats",result="success",quantile="0.99"} 0.000199063 node_exporter_scrape_duration_seconds_sum{collector="diskstats",result="success"} 0.0008093360000000001 node_exporter_scrape_duration_seconds_count{collector="diskstats",result="success"} 3 node_exporter_scrape_duration_seconds{collector="filefd",result="success",quantile="0.5"} 2.6501000000000003e-05 node_exporter_scrape_duration_seconds{collector="filefd",result="success",quantile="0.9"} 0.000149764 node_exporter_scrape_duration_seconds{collector="filefd",result="success",quantile="0.99"} 0.000149764 node_exporter_scrape_duration_seconds_sum{collector="filefd",result="success"} 0.0004463360000000001 node_exporter_scrape_duration_seconds_count{collector="filefd",result="success"} 3 node_exporter_scrape_duration_seconds{collector="filesystem",result="success",quantile="0.5"} 0.000350556 node_exporter_scrape_duration_seconds{collector="filesystem",result="success",quantile="0.9"} 0.000661542 node_exporter_scrape_duration_seconds{collector="filesystem",result="success",quantile="0.99"} 0.000661542 node_exporter_scrape_duration_seconds_sum{collector="filesystem",result="success"} 0.006669826000000001 node_exporter_scrape_duration_seconds_count{collector="filesystem",result="success"} 3 node_exporter_scrape_duration_seconds{collector="loadavg",result="success",quantile="0.5"} 4.1976e-05 node_exporter_scrape_duration_seconds{collector="loadavg",result="success",quantile="0.9"} 4.2433000000000005e-05 node_exporter_scrape_duration_seconds{collector="loadavg",result="success",quantile="0.99"} 4.2433000000000005e-05 node_exporter_scrape_duration_seconds_sum{collector="loadavg",result="success"} 0.0006603080000000001 node_exporter_scrape_duration_seconds_count{collector="loadavg",result="success"} 3 node_exporter_scrape_duration_seconds{collector="meminfo",result="success",quantile="0.5"} 0.000244919 node_exporter_scrape_duration_seconds{collector="meminfo",result="success",quantile="0.9"} 0.00038740000000000004 node_exporter_scrape_duration_seconds{collector="meminfo",result="success",quantile="0.99"} 0.00038740000000000004 node_exporter_scrape_duration_seconds_sum{collector="meminfo",result="success"} 0.003157154 node_exporter_scrape_duration_seconds_count{collector="meminfo",result="success"} 3 node_exporter_scrape_duration_seconds{collector="netdev",result="success",quantile="0.5"} 0.000152886 node_exporter_scrape_duration_seconds{collector="netdev",result="success",quantile="0.9"} 0.00048569400000000006 node_exporter_scrape_duration_seconds{collector="netdev",result="success",quantile="0.99"} 0.00048569400000000006 node_exporter_scrape_duration_seconds_sum{collector="netdev",result="success"} 0.0033770220000000004 node_exporter_scrape_duration_seconds_count{collector="netdev",result="success"} 3 node_exporter_scrape_duration_seconds{collector="netstat",result="success",quantile="0.5"} 0.0007874710000000001 node_exporter_scrape_duration_seconds{collector="netstat",result="success",quantile="0.9"} 0.001612906 node_exporter_scrape_duration_seconds{collector="netstat",result="success",quantile="0.99"} 0.001612906 node_exporter_scrape_duration_seconds_sum{collector="netstat",result="success"} 0.0055138800000000005 node_exporter_scrape_duration_seconds_count{collector="netstat",result="success"} 3 node_exporter_scrape_duration_seconds{collector="stat",result="success",quantile="0.5"} 9.3368e-05 node_exporter_scrape_duration_seconds{collector="stat",result="success",quantile="0.9"} 0.00014552100000000002 node_exporter_scrape_duration_seconds{collector="stat",result="success",quantile="0.99"} 0.00014552100000000002 node_exporter_scrape_duration_seconds_sum{collector="stat",result="success"} 0.00039008900000000004 node_exporter_scrape_duration_seconds_count{collector="stat",result="success"} 3 node_exporter_scrape_duration_seconds{collector="time",result="success",quantile="0.5"} 1.0485e-05 node_exporter_scrape_duration_seconds{collector="time",result="success",quantile="0.9"} 2.462e-05 node_exporter_scrape_duration_seconds{collector="time",result="success",quantile="0.99"} 2.462e-05 node_exporter_scrape_duration_seconds_sum{collector="time",result="success"} 6.4423e-05 node_exporter_scrape_duration_seconds_count{collector="time",result="success"} 3 node_exporter_scrape_duration_seconds{collector="uname",result="success",quantile="0.5"} 1.811e-05 node_exporter_scrape_duration_seconds{collector="uname",result="success",quantile="0.9"} 6.731300000000001e-05 node_exporter_scrape_duration_seconds{collector="uname",result="success",quantile="0.99"} 6.731300000000001e-05 node_exporter_scrape_duration_seconds_sum{collector="uname",result="success"} 0.00030852200000000004 node_exporter_scrape_duration_seconds_count{collector="uname",result="success"} 3 node_exporter_scrape_duration_seconds{collector="vmstat",result="success",quantile="0.5"} 0.00035561100000000003 node_exporter_scrape_duration_seconds{collector="vmstat",result="success",quantile="0.9"} 0.00046660900000000004 node_exporter_scrape_duration_seconds{collector="vmstat",result="success",quantile="0.99"} 0.00046660900000000004 node_exporter_scrape_duration_seconds_sum{collector="vmstat",result="success"} 0.002186003 node_exporter_scrape_duration_seconds_count{collector="vmstat",result="success"} 3 # HELP node_filefd_allocated File descriptor statistics: allocated. # TYPE node_filefd_allocated gauge node_filefd_allocated 1696 # HELP node_filefd_maximum File descriptor statistics: maximum. # TYPE node_filefd_maximum gauge node_filefd_maximum 382428 # HELP node_filesystem_avail Filesystem space available to non-root users in bytes. # TYPE node_filesystem_avail gauge node_filesystem_avail{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 2.977058816e+09 node_filesystem_avail{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 2.3885000704e+10 node_filesystem_avail{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 6.520070144e+09 node_filesystem_avail{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 8.374427648e+09 node_filesystem_avail{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 6.386853888e+10 node_filesystem_avail{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 1.37248768e+08 node_filesystem_avail{device="rootfs",fstype="rootfs",mountpoint="/"} 2.977058816e+09 node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 1.978773504e+09 node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 3.975168e+08 node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 3.975168e+08 # HELP node_filesystem_files Filesystem total file nodes. # TYPE node_filesystem_files gauge node_filesystem_files{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 6.991872e+06 node_filesystem_files{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 2.9360128e+07 node_filesystem_files{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 8.388608e+06 node_filesystem_files{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 4.194304e+06 node_filesystem_files{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 3.145728e+07 node_filesystem_files{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 512000 node_filesystem_files{device="rootfs",fstype="rootfs",mountpoint="/"} 6.991872e+06 node_filesystem_files{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 485249 node_filesystem_files{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 485249 node_filesystem_files{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 485249 # HELP node_filesystem_files_free Filesystem total free file nodes. # TYPE node_filesystem_files_free gauge node_filesystem_files_free{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 6.839777e+06 node_filesystem_files_free{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 2.9273287e+07 node_filesystem_files_free{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 8.376436e+06 node_filesystem_files_free{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 4.194188e+06 node_filesystem_files_free{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 3.1457181e+07 node_filesystem_files_free{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 511638 node_filesystem_files_free{device="rootfs",fstype="rootfs",mountpoint="/"} 6.839777e+06 node_filesystem_files_free{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 484725 node_filesystem_files_free{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 485248 node_filesystem_files_free{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 485248 # HELP node_filesystem_free Filesystem free space in bytes. # TYPE node_filesystem_free gauge node_filesystem_free{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 2.977058816e+09 node_filesystem_free{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 2.3885000704e+10 node_filesystem_free{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 6.520070144e+09 node_filesystem_free{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 8.374427648e+09 node_filesystem_free{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 6.386853888e+10 node_filesystem_free{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 1.37248768e+08 node_filesystem_free{device="rootfs",fstype="rootfs",mountpoint="/"} 2.977058816e+09 node_filesystem_free{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 1.978773504e+09 node_filesystem_free{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 3.975168e+08 node_filesystem_free{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 3.975168e+08 # HELP node_filesystem_readonly Filesystem read-only status. # TYPE node_filesystem_readonly gauge node_filesystem_readonly{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 0 node_filesystem_readonly{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 0 node_filesystem_readonly{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 0 node_filesystem_readonly{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 0 node_filesystem_readonly{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 0 node_filesystem_readonly{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 0 node_filesystem_readonly{device="rootfs",fstype="rootfs",mountpoint="/"} 0 node_filesystem_readonly{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 0 node_filesystem_readonly{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 0 node_filesystem_readonly{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 0 # HELP node_filesystem_size Filesystem size in bytes. # TYPE node_filesystem_size gauge node_filesystem_size{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 7.149191168e+09 node_filesystem_size{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 3.0054285312e+10 node_filesystem_size{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 8.579448832e+09 node_filesystem_size{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 8.579448832e+09 node_filesystem_size{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 6.439305216e+10 node_filesystem_size{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 5.20794112e+08 node_filesystem_size{device="rootfs",fstype="rootfs",mountpoint="/"} 7.149191168e+09 node_filesystem_size{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 1.987579904e+09 node_filesystem_size{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 3.975168e+08 node_filesystem_size{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 3.975168e+08 # HELP node_forks Total number of forks. # TYPE node_forks counter node_forks 3928 # HELP node_intr Total number of interrupts serviced. # TYPE node_intr counter node_intr 4.629898e+06 # HELP node_load1 1m load average. # TYPE node_load1 gauge node_load1 0.2 # HELP node_load15 15m load average. # TYPE node_load15 gauge node_load15 0.21 # HELP node_load5 5m load average. # TYPE node_load5 gauge node_load5 0.22 # HELP node_memory_Active Memory information field Active. # TYPE node_memory_Active gauge node_memory_Active 4.80772096e+08 # HELP node_memory_Active_anon Memory information field Active_anon. # TYPE node_memory_Active_anon gauge node_memory_Active_anon 3.62610688e+08 # HELP node_memory_Active_file Memory information field Active_file. # TYPE node_memory_Active_file gauge node_memory_Active_file 1.18161408e+08 # HELP node_memory_AnonHugePages Memory information field AnonHugePages. # TYPE node_memory_AnonHugePages gauge node_memory_AnonHugePages 1.2582912e+07 # HELP node_memory_AnonPages Memory information field AnonPages. # TYPE node_memory_AnonPages gauge node_memory_AnonPages 3.62438656e+08 # HELP node_memory_Bounce Memory information field Bounce. # TYPE node_memory_Bounce gauge node_memory_Bounce 0 # HELP node_memory_Buffers Memory information field Buffers. # TYPE node_memory_Buffers gauge node_memory_Buffers 1.90464e+06 # HELP node_memory_Cached Memory information field Cached. # TYPE node_memory_Cached gauge node_memory_Cached 3.12205312e+08 # HELP node_memory_CommitLimit Memory information field CommitLimit. # TYPE node_memory_CommitLimit gauge node_memory_CommitLimit 2.847408128e+09 # HELP node_memory_Committed_AS Memory information field Committed_AS. # TYPE node_memory_Committed_AS gauge node_memory_Committed_AS 5.08229632e+09 # HELP node_memory_DirectMap2M Memory information field DirectMap2M. # TYPE node_memory_DirectMap2M gauge node_memory_DirectMap2M 4.22576128e+09 # HELP node_memory_DirectMap4k Memory information field DirectMap4k. # TYPE node_memory_DirectMap4k gauge node_memory_DirectMap4k 6.914048e+07 # HELP node_memory_Dirty Memory information field Dirty. # TYPE node_memory_Dirty gauge node_memory_Dirty 69632 # HELP node_memory_HardwareCorrupted Memory information field HardwareCorrupted. # TYPE node_memory_HardwareCorrupted gauge node_memory_HardwareCorrupted 0 # HELP node_memory_HugePages_Free Memory information field HugePages_Free. # TYPE node_memory_HugePages_Free gauge node_memory_HugePages_Free 0 # HELP node_memory_HugePages_Rsvd Memory information field HugePages_Rsvd. # TYPE node_memory_HugePages_Rsvd gauge node_memory_HugePages_Rsvd 0 # HELP node_memory_HugePages_Surp Memory information field HugePages_Surp. # TYPE node_memory_HugePages_Surp gauge node_memory_HugePages_Surp 0 # HELP node_memory_HugePages_Total Memory information field HugePages_Total. # TYPE node_memory_HugePages_Total gauge node_memory_HugePages_Total 0 # HELP node_memory_Hugepagesize Memory information field Hugepagesize. # TYPE node_memory_Hugepagesize gauge node_memory_Hugepagesize 2.097152e+06 # HELP node_memory_Inactive Memory information field Inactive. # TYPE node_memory_Inactive gauge node_memory_Inactive 1.95702784e+08 # HELP node_memory_Inactive_anon Memory information field Inactive_anon. # TYPE node_memory_Inactive_anon gauge node_memory_Inactive_anon 8.564736e+06 # HELP node_memory_Inactive_file Memory information field Inactive_file. # TYPE node_memory_Inactive_file gauge node_memory_Inactive_file 1.87138048e+08 # HELP node_memory_KernelStack Memory information field KernelStack. # TYPE node_memory_KernelStack gauge node_memory_KernelStack 1.1583488e+07 # HELP node_memory_Mapped Memory information field Mapped. # TYPE node_memory_Mapped gauge node_memory_Mapped 6.7530752e+07 # HELP node_memory_MemAvailable Memory information field MemAvailable. # TYPE node_memory_MemAvailable gauge node_memory_MemAvailable 3.273515008e+09 # HELP node_memory_MemFree Memory information field MemFree. # TYPE node_memory_MemFree gauge node_memory_MemFree 3.15346944e+09 # HELP node_memory_MemTotal Memory information field MemTotal. # TYPE node_memory_MemTotal gauge node_memory_MemTotal 3.975163904e+09 # HELP node_memory_Mlocked Memory information field Mlocked. # TYPE node_memory_Mlocked gauge node_memory_Mlocked 0 # HELP node_memory_NFS_Unstable Memory information field NFS_Unstable. # TYPE node_memory_NFS_Unstable gauge node_memory_NFS_Unstable 0 # HELP node_memory_PageTables Memory information field PageTables. # TYPE node_memory_PageTables gauge node_memory_PageTables 9.105408e+06 # HELP node_memory_SReclaimable Memory information field SReclaimable. # TYPE node_memory_SReclaimable gauge node_memory_SReclaimable 4.4638208e+07 # HELP node_memory_SUnreclaim Memory information field SUnreclaim. # TYPE node_memory_SUnreclaim gauge node_memory_SUnreclaim 3.2059392e+07 # HELP node_memory_Shmem Memory information field Shmem. # TYPE node_memory_Shmem gauge node_memory_Shmem 8.810496e+06 # HELP node_memory_Slab Memory information field Slab. # TYPE node_memory_Slab gauge node_memory_Slab 7.66976e+07 # HELP node_memory_SwapCached Memory information field SwapCached. # TYPE node_memory_SwapCached gauge node_memory_SwapCached 0 # HELP node_memory_SwapFree Memory information field SwapFree. # TYPE node_memory_SwapFree gauge node_memory_SwapFree 8.59828224e+08 # HELP node_memory_SwapTotal Memory information field SwapTotal. # TYPE node_memory_SwapTotal gauge node_memory_SwapTotal 8.59828224e+08 # HELP node_memory_Unevictable Memory information field Unevictable. # TYPE node_memory_Unevictable gauge node_memory_Unevictable 0 # HELP node_memory_VmallocChunk Memory information field VmallocChunk. # TYPE node_memory_VmallocChunk gauge node_memory_VmallocChunk 3.5184346918912e+13 # HELP node_memory_VmallocTotal Memory information field VmallocTotal. # TYPE node_memory_VmallocTotal gauge node_memory_VmallocTotal 3.5184372087808e+13 # HELP node_memory_VmallocUsed Memory information field VmallocUsed. # TYPE node_memory_VmallocUsed gauge node_memory_VmallocUsed 1.6748544e+07 # HELP node_memory_Writeback Memory information field Writeback. # TYPE node_memory_Writeback gauge node_memory_Writeback 0 # HELP node_memory_WritebackTmp Memory information field WritebackTmp. # TYPE node_memory_WritebackTmp gauge node_memory_WritebackTmp 0 # HELP node_netstat_IcmpMsg_InType3 Protocol IcmpMsg statistic InType3. # TYPE node_netstat_IcmpMsg_InType3 untyped node_netstat_IcmpMsg_InType3 32 # HELP node_netstat_IcmpMsg_OutType3 Protocol IcmpMsg statistic OutType3. # TYPE node_netstat_IcmpMsg_OutType3 untyped node_netstat_IcmpMsg_OutType3 32 # HELP node_netstat_Icmp_InAddrMaskReps Protocol Icmp statistic InAddrMaskReps. # TYPE node_netstat_Icmp_InAddrMaskReps untyped node_netstat_Icmp_InAddrMaskReps 0 # HELP node_netstat_Icmp_InAddrMasks Protocol Icmp statistic InAddrMasks. # TYPE node_netstat_Icmp_InAddrMasks untyped node_netstat_Icmp_InAddrMasks 0 # HELP node_netstat_Icmp_InCsumErrors Protocol Icmp statistic InCsumErrors. # TYPE node_netstat_Icmp_InCsumErrors untyped node_netstat_Icmp_InCsumErrors 0 # HELP node_netstat_Icmp_InDestUnreachs Protocol Icmp statistic InDestUnreachs. # TYPE node_netstat_Icmp_InDestUnreachs untyped node_netstat_Icmp_InDestUnreachs 32 # HELP node_netstat_Icmp_InEchoReps Protocol Icmp statistic InEchoReps. # TYPE node_netstat_Icmp_InEchoReps untyped node_netstat_Icmp_InEchoReps 0 # HELP node_netstat_Icmp_InEchos Protocol Icmp statistic InEchos. # TYPE node_netstat_Icmp_InEchos untyped node_netstat_Icmp_InEchos 0 # HELP node_netstat_Icmp_InErrors Protocol Icmp statistic InErrors. # TYPE node_netstat_Icmp_InErrors untyped node_netstat_Icmp_InErrors 0 # HELP node_netstat_Icmp_InMsgs Protocol Icmp statistic InMsgs. # TYPE node_netstat_Icmp_InMsgs untyped node_netstat_Icmp_InMsgs 32 # HELP node_netstat_Icmp_InParmProbs Protocol Icmp statistic InParmProbs. # TYPE node_netstat_Icmp_InParmProbs untyped node_netstat_Icmp_InParmProbs 0 # HELP node_netstat_Icmp_InRedirects Protocol Icmp statistic InRedirects. # TYPE node_netstat_Icmp_InRedirects untyped node_netstat_Icmp_InRedirects 0 # HELP node_netstat_Icmp_InSrcQuenchs Protocol Icmp statistic InSrcQuenchs. # TYPE node_netstat_Icmp_InSrcQuenchs untyped node_netstat_Icmp_InSrcQuenchs 0 # HELP node_netstat_Icmp_InTimeExcds Protocol Icmp statistic InTimeExcds. # TYPE node_netstat_Icmp_InTimeExcds untyped node_netstat_Icmp_InTimeExcds 0 # HELP node_netstat_Icmp_InTimestampReps Protocol Icmp statistic InTimestampReps. # TYPE node_netstat_Icmp_InTimestampReps untyped node_netstat_Icmp_InTimestampReps 0 # HELP node_netstat_Icmp_InTimestamps Protocol Icmp statistic InTimestamps. # TYPE node_netstat_Icmp_InTimestamps untyped node_netstat_Icmp_InTimestamps 0 # HELP node_netstat_Icmp_OutAddrMaskReps Protocol Icmp statistic OutAddrMaskReps. # TYPE node_netstat_Icmp_OutAddrMaskReps untyped node_netstat_Icmp_OutAddrMaskReps 0 # HELP node_netstat_Icmp_OutAddrMasks Protocol Icmp statistic OutAddrMasks. # TYPE node_netstat_Icmp_OutAddrMasks untyped node_netstat_Icmp_OutAddrMasks 0 # HELP node_netstat_Icmp_OutDestUnreachs Protocol Icmp statistic OutDestUnreachs. # TYPE node_netstat_Icmp_OutDestUnreachs untyped node_netstat_Icmp_OutDestUnreachs 32 # HELP node_netstat_Icmp_OutEchoReps Protocol Icmp statistic OutEchoReps. # TYPE node_netstat_Icmp_OutEchoReps untyped node_netstat_Icmp_OutEchoReps 0 # HELP node_netstat_Icmp_OutEchos Protocol Icmp statistic OutEchos. # TYPE node_netstat_Icmp_OutEchos untyped node_netstat_Icmp_OutEchos 0 # HELP node_netstat_Icmp_OutErrors Protocol Icmp statistic OutErrors. # TYPE node_netstat_Icmp_OutErrors untyped node_netstat_Icmp_OutErrors 0 # HELP node_netstat_Icmp_OutMsgs Protocol Icmp statistic OutMsgs. # TYPE node_netstat_Icmp_OutMsgs untyped node_netstat_Icmp_OutMsgs 32 # HELP node_netstat_Icmp_OutParmProbs Protocol Icmp statistic OutParmProbs. # TYPE node_netstat_Icmp_OutParmProbs untyped node_netstat_Icmp_OutParmProbs 0 # HELP node_netstat_Icmp_OutRedirects Protocol Icmp statistic OutRedirects. # TYPE node_netstat_Icmp_OutRedirects untyped node_netstat_Icmp_OutRedirects 0 # HELP node_netstat_Icmp_OutSrcQuenchs Protocol Icmp statistic OutSrcQuenchs. # TYPE node_netstat_Icmp_OutSrcQuenchs untyped node_netstat_Icmp_OutSrcQuenchs 0 # HELP node_netstat_Icmp_OutTimeExcds Protocol Icmp statistic OutTimeExcds. # TYPE node_netstat_Icmp_OutTimeExcds untyped node_netstat_Icmp_OutTimeExcds 0 # HELP node_netstat_Icmp_OutTimestampReps Protocol Icmp statistic OutTimestampReps. # TYPE node_netstat_Icmp_OutTimestampReps untyped node_netstat_Icmp_OutTimestampReps 0 # HELP node_netstat_Icmp_OutTimestamps Protocol Icmp statistic OutTimestamps. # TYPE node_netstat_Icmp_OutTimestamps untyped node_netstat_Icmp_OutTimestamps 0 # HELP node_netstat_IpExt_InBcastOctets Protocol IpExt statistic InBcastOctets. # TYPE node_netstat_IpExt_InBcastOctets untyped node_netstat_IpExt_InBcastOctets 16200 # HELP node_netstat_IpExt_InBcastPkts Protocol IpExt statistic InBcastPkts. # TYPE node_netstat_IpExt_InBcastPkts untyped node_netstat_IpExt_InBcastPkts 225 # HELP node_netstat_IpExt_InCEPkts Protocol IpExt statistic InCEPkts. # TYPE node_netstat_IpExt_InCEPkts untyped node_netstat_IpExt_InCEPkts 0 # HELP node_netstat_IpExt_InCsumErrors Protocol IpExt statistic InCsumErrors. # TYPE node_netstat_IpExt_InCsumErrors untyped node_netstat_IpExt_InCsumErrors 0 # HELP node_netstat_IpExt_InECT0Pkts Protocol IpExt statistic InECT0Pkts. # TYPE node_netstat_IpExt_InECT0Pkts untyped node_netstat_IpExt_InECT0Pkts 0 # HELP node_netstat_IpExt_InECT1Pkts Protocol IpExt statistic InECT1Pkts. # TYPE node_netstat_IpExt_InECT1Pkts untyped node_netstat_IpExt_InECT1Pkts 0 # HELP node_netstat_IpExt_InMcastOctets Protocol IpExt statistic InMcastOctets. # TYPE node_netstat_IpExt_InMcastOctets untyped node_netstat_IpExt_InMcastOctets 0 # HELP node_netstat_IpExt_InMcastPkts Protocol IpExt statistic InMcastPkts. # TYPE node_netstat_IpExt_InMcastPkts untyped node_netstat_IpExt_InMcastPkts 0 # HELP node_netstat_IpExt_InNoECTPkts Protocol IpExt statistic InNoECTPkts. # TYPE node_netstat_IpExt_InNoECTPkts untyped node_netstat_IpExt_InNoECTPkts 139536 # HELP node_netstat_IpExt_InNoRoutes Protocol IpExt statistic InNoRoutes. # TYPE node_netstat_IpExt_InNoRoutes untyped node_netstat_IpExt_InNoRoutes 1 # HELP node_netstat_IpExt_InOctets Protocol IpExt statistic InOctets. # TYPE node_netstat_IpExt_InOctets untyped node_netstat_IpExt_InOctets 3.4550209e+07 # HELP node_netstat_IpExt_InTruncatedPkts Protocol IpExt statistic InTruncatedPkts. # TYPE node_netstat_IpExt_InTruncatedPkts untyped node_netstat_IpExt_InTruncatedPkts 0 # HELP node_netstat_IpExt_OutBcastOctets Protocol IpExt statistic OutBcastOctets. # TYPE node_netstat_IpExt_OutBcastOctets untyped node_netstat_IpExt_OutBcastOctets 0 # HELP node_netstat_IpExt_OutBcastPkts Protocol IpExt statistic OutBcastPkts. # TYPE node_netstat_IpExt_OutBcastPkts untyped node_netstat_IpExt_OutBcastPkts 0 # HELP node_netstat_IpExt_OutMcastOctets Protocol IpExt statistic OutMcastOctets. # TYPE node_netstat_IpExt_OutMcastOctets untyped node_netstat_IpExt_OutMcastOctets 0 # HELP node_netstat_IpExt_OutMcastPkts Protocol IpExt statistic OutMcastPkts. # TYPE node_netstat_IpExt_OutMcastPkts untyped node_netstat_IpExt_OutMcastPkts 0 # HELP node_netstat_IpExt_OutOctets Protocol IpExt statistic OutOctets. # TYPE node_netstat_IpExt_OutOctets untyped node_netstat_IpExt_OutOctets 3.469735e+07 # HELP node_netstat_Ip_DefaultTTL Protocol Ip statistic DefaultTTL. # TYPE node_netstat_Ip_DefaultTTL untyped node_netstat_Ip_DefaultTTL 64 # HELP node_netstat_Ip_ForwDatagrams Protocol Ip statistic ForwDatagrams. # TYPE node_netstat_Ip_ForwDatagrams untyped node_netstat_Ip_ForwDatagrams 0 # HELP node_netstat_Ip_Forwarding Protocol Ip statistic Forwarding. # TYPE node_netstat_Ip_Forwarding untyped node_netstat_Ip_Forwarding 1 # HELP node_netstat_Ip_FragCreates Protocol Ip statistic FragCreates. # TYPE node_netstat_Ip_FragCreates untyped node_netstat_Ip_FragCreates 0 # HELP node_netstat_Ip_FragFails Protocol Ip statistic FragFails. # TYPE node_netstat_Ip_FragFails untyped node_netstat_Ip_FragFails 0 # HELP node_netstat_Ip_FragOKs Protocol Ip statistic FragOKs. # TYPE node_netstat_Ip_FragOKs untyped node_netstat_Ip_FragOKs 0 # HELP node_netstat_Ip_InAddrErrors Protocol Ip statistic InAddrErrors. # TYPE node_netstat_Ip_InAddrErrors untyped node_netstat_Ip_InAddrErrors 0 # HELP node_netstat_Ip_InDelivers Protocol Ip statistic InDelivers. # TYPE node_netstat_Ip_InDelivers untyped node_netstat_Ip_InDelivers 139535 # HELP node_netstat_Ip_InDiscards Protocol Ip statistic InDiscards. # TYPE node_netstat_Ip_InDiscards untyped node_netstat_Ip_InDiscards 0 # HELP node_netstat_Ip_InHdrErrors Protocol Ip statistic InHdrErrors. # TYPE node_netstat_Ip_InHdrErrors untyped node_netstat_Ip_InHdrErrors 0 # HELP node_netstat_Ip_InReceives Protocol Ip statistic InReceives. # TYPE node_netstat_Ip_InReceives untyped node_netstat_Ip_InReceives 139536 # HELP node_netstat_Ip_InUnknownProtos Protocol Ip statistic InUnknownProtos. # TYPE node_netstat_Ip_InUnknownProtos untyped node_netstat_Ip_InUnknownProtos 0 # HELP node_netstat_Ip_OutDiscards Protocol Ip statistic OutDiscards. # TYPE node_netstat_Ip_OutDiscards untyped node_netstat_Ip_OutDiscards 16 # HELP node_netstat_Ip_OutNoRoutes Protocol Ip statistic OutNoRoutes. # TYPE node_netstat_Ip_OutNoRoutes untyped node_netstat_Ip_OutNoRoutes 0 # HELP node_netstat_Ip_OutRequests Protocol Ip statistic OutRequests. # TYPE node_netstat_Ip_OutRequests untyped node_netstat_Ip_OutRequests 139089 # HELP node_netstat_Ip_ReasmFails Protocol Ip statistic ReasmFails. # TYPE node_netstat_Ip_ReasmFails untyped node_netstat_Ip_ReasmFails 0 # HELP node_netstat_Ip_ReasmOKs Protocol Ip statistic ReasmOKs. # TYPE node_netstat_Ip_ReasmOKs untyped node_netstat_Ip_ReasmOKs 0 # HELP node_netstat_Ip_ReasmReqds Protocol Ip statistic ReasmReqds. # TYPE node_netstat_Ip_ReasmReqds untyped node_netstat_Ip_ReasmReqds 0 # HELP node_netstat_Ip_ReasmTimeout Protocol Ip statistic ReasmTimeout. # TYPE node_netstat_Ip_ReasmTimeout untyped node_netstat_Ip_ReasmTimeout 0 # HELP node_netstat_TcpExt_ArpFilter Protocol TcpExt statistic ArpFilter. # TYPE node_netstat_TcpExt_ArpFilter untyped node_netstat_TcpExt_ArpFilter 0 # HELP node_netstat_TcpExt_BusyPollRxPackets Protocol TcpExt statistic BusyPollRxPackets. # TYPE node_netstat_TcpExt_BusyPollRxPackets untyped node_netstat_TcpExt_BusyPollRxPackets 0 # HELP node_netstat_TcpExt_DelayedACKLocked Protocol TcpExt statistic DelayedACKLocked. # TYPE node_netstat_TcpExt_DelayedACKLocked untyped node_netstat_TcpExt_DelayedACKLocked 0 # HELP node_netstat_TcpExt_DelayedACKLost Protocol TcpExt statistic DelayedACKLost. # TYPE node_netstat_TcpExt_DelayedACKLost untyped node_netstat_TcpExt_DelayedACKLost 0 # HELP node_netstat_TcpExt_DelayedACKs Protocol TcpExt statistic DelayedACKs. # TYPE node_netstat_TcpExt_DelayedACKs untyped node_netstat_TcpExt_DelayedACKs 3374 # HELP node_netstat_TcpExt_EmbryonicRsts Protocol TcpExt statistic EmbryonicRsts. # TYPE node_netstat_TcpExt_EmbryonicRsts untyped node_netstat_TcpExt_EmbryonicRsts 0 # HELP node_netstat_TcpExt_IPReversePathFilter Protocol TcpExt statistic IPReversePathFilter. # TYPE node_netstat_TcpExt_IPReversePathFilter untyped node_netstat_TcpExt_IPReversePathFilter 0 # HELP node_netstat_TcpExt_ListenDrops Protocol TcpExt statistic ListenDrops. # TYPE node_netstat_TcpExt_ListenDrops untyped node_netstat_TcpExt_ListenDrops 0 # HELP node_netstat_TcpExt_ListenOverflows Protocol TcpExt statistic ListenOverflows. # TYPE node_netstat_TcpExt_ListenOverflows untyped node_netstat_TcpExt_ListenOverflows 0 # HELP node_netstat_TcpExt_LockDroppedIcmps Protocol TcpExt statistic LockDroppedIcmps. # TYPE node_netstat_TcpExt_LockDroppedIcmps untyped node_netstat_TcpExt_LockDroppedIcmps 0 # HELP node_netstat_TcpExt_OfoPruned Protocol TcpExt statistic OfoPruned. # TYPE node_netstat_TcpExt_OfoPruned untyped node_netstat_TcpExt_OfoPruned 0 # HELP node_netstat_TcpExt_OutOfWindowIcmps Protocol TcpExt statistic OutOfWindowIcmps. # TYPE node_netstat_TcpExt_OutOfWindowIcmps untyped node_netstat_TcpExt_OutOfWindowIcmps 0 # HELP node_netstat_TcpExt_PAWSActive Protocol TcpExt statistic PAWSActive. # TYPE node_netstat_TcpExt_PAWSActive untyped node_netstat_TcpExt_PAWSActive 0 # HELP node_netstat_TcpExt_PAWSEstab Protocol TcpExt statistic PAWSEstab. # TYPE node_netstat_TcpExt_PAWSEstab untyped node_netstat_TcpExt_PAWSEstab 0 # HELP node_netstat_TcpExt_PAWSPassive Protocol TcpExt statistic PAWSPassive. # TYPE node_netstat_TcpExt_PAWSPassive untyped node_netstat_TcpExt_PAWSPassive 0 # HELP node_netstat_TcpExt_PruneCalled Protocol TcpExt statistic PruneCalled. # TYPE node_netstat_TcpExt_PruneCalled untyped node_netstat_TcpExt_PruneCalled 0 # HELP node_netstat_TcpExt_RcvPruned Protocol TcpExt statistic RcvPruned. # TYPE node_netstat_TcpExt_RcvPruned untyped node_netstat_TcpExt_RcvPruned 0 # HELP node_netstat_TcpExt_SyncookiesFailed Protocol TcpExt statistic SyncookiesFailed. # TYPE node_netstat_TcpExt_SyncookiesFailed untyped node_netstat_TcpExt_SyncookiesFailed 0 # HELP node_netstat_TcpExt_SyncookiesRecv Protocol TcpExt statistic SyncookiesRecv. # TYPE node_netstat_TcpExt_SyncookiesRecv untyped node_netstat_TcpExt_SyncookiesRecv 0 # HELP node_netstat_TcpExt_SyncookiesSent Protocol TcpExt statistic SyncookiesSent. # TYPE node_netstat_TcpExt_SyncookiesSent untyped node_netstat_TcpExt_SyncookiesSent 0 # HELP node_netstat_TcpExt_TCPACKSkippedChallenge Protocol TcpExt statistic TCPACKSkippedChallenge. # TYPE node_netstat_TcpExt_TCPACKSkippedChallenge untyped node_netstat_TcpExt_TCPACKSkippedChallenge 0 # HELP node_netstat_TcpExt_TCPACKSkippedFinWait2 Protocol TcpExt statistic TCPACKSkippedFinWait2. # TYPE node_netstat_TcpExt_TCPACKSkippedFinWait2 untyped node_netstat_TcpExt_TCPACKSkippedFinWait2 0 # HELP node_netstat_TcpExt_TCPACKSkippedPAWS Protocol TcpExt statistic TCPACKSkippedPAWS. # TYPE node_netstat_TcpExt_TCPACKSkippedPAWS untyped node_netstat_TcpExt_TCPACKSkippedPAWS 0 # HELP node_netstat_TcpExt_TCPACKSkippedSeq Protocol TcpExt statistic TCPACKSkippedSeq. # TYPE node_netstat_TcpExt_TCPACKSkippedSeq untyped node_netstat_TcpExt_TCPACKSkippedSeq 0 # HELP node_netstat_TcpExt_TCPACKSkippedSynRecv Protocol TcpExt statistic TCPACKSkippedSynRecv. # TYPE node_netstat_TcpExt_TCPACKSkippedSynRecv untyped node_netstat_TcpExt_TCPACKSkippedSynRecv 0 # HELP node_netstat_TcpExt_TCPACKSkippedTimeWait Protocol TcpExt statistic TCPACKSkippedTimeWait. # TYPE node_netstat_TcpExt_TCPACKSkippedTimeWait untyped node_netstat_TcpExt_TCPACKSkippedTimeWait 0 # HELP node_netstat_TcpExt_TCPAbortFailed Protocol TcpExt statistic TCPAbortFailed. # TYPE node_netstat_TcpExt_TCPAbortFailed untyped node_netstat_TcpExt_TCPAbortFailed 0 # HELP node_netstat_TcpExt_TCPAbortOnClose Protocol TcpExt statistic TCPAbortOnClose. # TYPE node_netstat_TcpExt_TCPAbortOnClose untyped node_netstat_TcpExt_TCPAbortOnClose 1 # HELP node_netstat_TcpExt_TCPAbortOnData Protocol TcpExt statistic TCPAbortOnData. # TYPE node_netstat_TcpExt_TCPAbortOnData untyped node_netstat_TcpExt_TCPAbortOnData 2 # HELP node_netstat_TcpExt_TCPAbortOnLinger Protocol TcpExt statistic TCPAbortOnLinger. # TYPE node_netstat_TcpExt_TCPAbortOnLinger untyped node_netstat_TcpExt_TCPAbortOnLinger 0 # HELP node_netstat_TcpExt_TCPAbortOnMemory Protocol TcpExt statistic TCPAbortOnMemory. # TYPE node_netstat_TcpExt_TCPAbortOnMemory untyped node_netstat_TcpExt_TCPAbortOnMemory 0 # HELP node_netstat_TcpExt_TCPAbortOnTimeout Protocol TcpExt statistic TCPAbortOnTimeout. # TYPE node_netstat_TcpExt_TCPAbortOnTimeout untyped node_netstat_TcpExt_TCPAbortOnTimeout 0 # HELP node_netstat_TcpExt_TCPAutoCorking Protocol TcpExt statistic TCPAutoCorking. # TYPE node_netstat_TcpExt_TCPAutoCorking untyped node_netstat_TcpExt_TCPAutoCorking 4 # HELP node_netstat_TcpExt_TCPBacklogDrop Protocol TcpExt statistic TCPBacklogDrop. # TYPE node_netstat_TcpExt_TCPBacklogDrop untyped node_netstat_TcpExt_TCPBacklogDrop 0 # HELP node_netstat_TcpExt_TCPChallengeACK Protocol TcpExt statistic TCPChallengeACK. # TYPE node_netstat_TcpExt_TCPChallengeACK untyped node_netstat_TcpExt_TCPChallengeACK 0 # HELP node_netstat_TcpExt_TCPDSACKIgnoredNoUndo Protocol TcpExt statistic TCPDSACKIgnoredNoUndo. # TYPE node_netstat_TcpExt_TCPDSACKIgnoredNoUndo untyped node_netstat_TcpExt_TCPDSACKIgnoredNoUndo 0 # HELP node_netstat_TcpExt_TCPDSACKIgnoredOld Protocol TcpExt statistic TCPDSACKIgnoredOld. # TYPE node_netstat_TcpExt_TCPDSACKIgnoredOld untyped node_netstat_TcpExt_TCPDSACKIgnoredOld 0 # HELP node_netstat_TcpExt_TCPDSACKOfoRecv Protocol TcpExt statistic TCPDSACKOfoRecv. # TYPE node_netstat_TcpExt_TCPDSACKOfoRecv untyped node_netstat_TcpExt_TCPDSACKOfoRecv 0 # HELP node_netstat_TcpExt_TCPDSACKOfoSent Protocol TcpExt statistic TCPDSACKOfoSent. # TYPE node_netstat_TcpExt_TCPDSACKOfoSent untyped node_netstat_TcpExt_TCPDSACKOfoSent 0 # HELP node_netstat_TcpExt_TCPDSACKOldSent Protocol TcpExt statistic TCPDSACKOldSent. # TYPE node_netstat_TcpExt_TCPDSACKOldSent untyped node_netstat_TcpExt_TCPDSACKOldSent 0 # HELP node_netstat_TcpExt_TCPDSACKRecv Protocol TcpExt statistic TCPDSACKRecv. # TYPE node_netstat_TcpExt_TCPDSACKRecv untyped node_netstat_TcpExt_TCPDSACKRecv 0 # HELP node_netstat_TcpExt_TCPDSACKUndo Protocol TcpExt statistic TCPDSACKUndo. # TYPE node_netstat_TcpExt_TCPDSACKUndo untyped node_netstat_TcpExt_TCPDSACKUndo 0 # HELP node_netstat_TcpExt_TCPDeferAcceptDrop Protocol TcpExt statistic TCPDeferAcceptDrop. # TYPE node_netstat_TcpExt_TCPDeferAcceptDrop untyped node_netstat_TcpExt_TCPDeferAcceptDrop 0 # HELP node_netstat_TcpExt_TCPDirectCopyFromBacklog Protocol TcpExt statistic TCPDirectCopyFromBacklog. # TYPE node_netstat_TcpExt_TCPDirectCopyFromBacklog untyped node_netstat_TcpExt_TCPDirectCopyFromBacklog 0 # HELP node_netstat_TcpExt_TCPDirectCopyFromPrequeue Protocol TcpExt statistic TCPDirectCopyFromPrequeue. # TYPE node_netstat_TcpExt_TCPDirectCopyFromPrequeue untyped node_netstat_TcpExt_TCPDirectCopyFromPrequeue 1.062e+06 # HELP node_netstat_TcpExt_TCPFACKReorder Protocol TcpExt statistic TCPFACKReorder. # TYPE node_netstat_TcpExt_TCPFACKReorder untyped node_netstat_TcpExt_TCPFACKReorder 0 # HELP node_netstat_TcpExt_TCPFastOpenActive Protocol TcpExt statistic TCPFastOpenActive. # TYPE node_netstat_TcpExt_TCPFastOpenActive untyped node_netstat_TcpExt_TCPFastOpenActive 0 # HELP node_netstat_TcpExt_TCPFastOpenActiveFail Protocol TcpExt statistic TCPFastOpenActiveFail. # TYPE node_netstat_TcpExt_TCPFastOpenActiveFail untyped node_netstat_TcpExt_TCPFastOpenActiveFail 0 # HELP node_netstat_TcpExt_TCPFastOpenCookieReqd Protocol TcpExt statistic TCPFastOpenCookieReqd. # TYPE node_netstat_TcpExt_TCPFastOpenCookieReqd untyped node_netstat_TcpExt_TCPFastOpenCookieReqd 0 # HELP node_netstat_TcpExt_TCPFastOpenListenOverflow Protocol TcpExt statistic TCPFastOpenListenOverflow. # TYPE node_netstat_TcpExt_TCPFastOpenListenOverflow untyped node_netstat_TcpExt_TCPFastOpenListenOverflow 0 # HELP node_netstat_TcpExt_TCPFastOpenPassive Protocol TcpExt statistic TCPFastOpenPassive. # TYPE node_netstat_TcpExt_TCPFastOpenPassive untyped node_netstat_TcpExt_TCPFastOpenPassive 0 # HELP node_netstat_TcpExt_TCPFastOpenPassiveFail Protocol TcpExt statistic TCPFastOpenPassiveFail. # TYPE node_netstat_TcpExt_TCPFastOpenPassiveFail untyped node_netstat_TcpExt_TCPFastOpenPassiveFail 0 # HELP node_netstat_TcpExt_TCPFastRetrans Protocol TcpExt statistic TCPFastRetrans. # TYPE node_netstat_TcpExt_TCPFastRetrans untyped node_netstat_TcpExt_TCPFastRetrans 0 # HELP node_netstat_TcpExt_TCPForwardRetrans Protocol TcpExt statistic TCPForwardRetrans. # TYPE node_netstat_TcpExt_TCPForwardRetrans untyped node_netstat_TcpExt_TCPForwardRetrans 0 # HELP node_netstat_TcpExt_TCPFromZeroWindowAdv Protocol TcpExt statistic TCPFromZeroWindowAdv. # TYPE node_netstat_TcpExt_TCPFromZeroWindowAdv untyped node_netstat_TcpExt_TCPFromZeroWindowAdv 0 # HELP node_netstat_TcpExt_TCPFullUndo Protocol TcpExt statistic TCPFullUndo. # TYPE node_netstat_TcpExt_TCPFullUndo untyped node_netstat_TcpExt_TCPFullUndo 0 # HELP node_netstat_TcpExt_TCPHPAcks Protocol TcpExt statistic TCPHPAcks. # TYPE node_netstat_TcpExt_TCPHPAcks untyped node_netstat_TcpExt_TCPHPAcks 21314 # HELP node_netstat_TcpExt_TCPHPHits Protocol TcpExt statistic TCPHPHits. # TYPE node_netstat_TcpExt_TCPHPHits untyped node_netstat_TcpExt_TCPHPHits 3607 # HELP node_netstat_TcpExt_TCPHPHitsToUser Protocol TcpExt statistic TCPHPHitsToUser. # TYPE node_netstat_TcpExt_TCPHPHitsToUser untyped node_netstat_TcpExt_TCPHPHitsToUser 0 # HELP node_netstat_TcpExt_TCPHystartDelayCwnd Protocol TcpExt statistic TCPHystartDelayCwnd. # TYPE node_netstat_TcpExt_TCPHystartDelayCwnd untyped node_netstat_TcpExt_TCPHystartDelayCwnd 0 # HELP node_netstat_TcpExt_TCPHystartDelayDetect Protocol TcpExt statistic TCPHystartDelayDetect. # TYPE node_netstat_TcpExt_TCPHystartDelayDetect untyped node_netstat_TcpExt_TCPHystartDelayDetect 0 # HELP node_netstat_TcpExt_TCPHystartTrainCwnd Protocol TcpExt statistic TCPHystartTrainCwnd. # TYPE node_netstat_TcpExt_TCPHystartTrainCwnd untyped node_netstat_TcpExt_TCPHystartTrainCwnd 0 # HELP node_netstat_TcpExt_TCPHystartTrainDetect Protocol TcpExt statistic TCPHystartTrainDetect. # TYPE node_netstat_TcpExt_TCPHystartTrainDetect untyped node_netstat_TcpExt_TCPHystartTrainDetect 0 # HELP node_netstat_TcpExt_TCPLossFailures Protocol TcpExt statistic TCPLossFailures. # TYPE node_netstat_TcpExt_TCPLossFailures untyped node_netstat_TcpExt_TCPLossFailures 0 # HELP node_netstat_TcpExt_TCPLossProbeRecovery Protocol TcpExt statistic TCPLossProbeRecovery. # TYPE node_netstat_TcpExt_TCPLossProbeRecovery untyped node_netstat_TcpExt_TCPLossProbeRecovery 0 # HELP node_netstat_TcpExt_TCPLossProbes Protocol TcpExt statistic TCPLossProbes. # TYPE node_netstat_TcpExt_TCPLossProbes untyped node_netstat_TcpExt_TCPLossProbes 0 # HELP node_netstat_TcpExt_TCPLossUndo Protocol TcpExt statistic TCPLossUndo. # TYPE node_netstat_TcpExt_TCPLossUndo untyped node_netstat_TcpExt_TCPLossUndo 0 # HELP node_netstat_TcpExt_TCPLostRetransmit Protocol TcpExt statistic TCPLostRetransmit. # TYPE node_netstat_TcpExt_TCPLostRetransmit untyped node_netstat_TcpExt_TCPLostRetransmit 0 # HELP node_netstat_TcpExt_TCPMD5NotFound Protocol TcpExt statistic TCPMD5NotFound. # TYPE node_netstat_TcpExt_TCPMD5NotFound untyped node_netstat_TcpExt_TCPMD5NotFound 0 # HELP node_netstat_TcpExt_TCPMD5Unexpected Protocol TcpExt statistic TCPMD5Unexpected. # TYPE node_netstat_TcpExt_TCPMD5Unexpected untyped node_netstat_TcpExt_TCPMD5Unexpected 0 # HELP node_netstat_TcpExt_TCPMemoryPressures Protocol TcpExt statistic TCPMemoryPressures. # TYPE node_netstat_TcpExt_TCPMemoryPressures untyped node_netstat_TcpExt_TCPMemoryPressures 0 # HELP node_netstat_TcpExt_TCPMinTTLDrop Protocol TcpExt statistic TCPMinTTLDrop. # TYPE node_netstat_TcpExt_TCPMinTTLDrop untyped node_netstat_TcpExt_TCPMinTTLDrop 0 # HELP node_netstat_TcpExt_TCPOFODrop Protocol TcpExt statistic TCPOFODrop. # TYPE node_netstat_TcpExt_TCPOFODrop untyped node_netstat_TcpExt_TCPOFODrop 0 # HELP node_netstat_TcpExt_TCPOFOMerge Protocol TcpExt statistic TCPOFOMerge. # TYPE node_netstat_TcpExt_TCPOFOMerge untyped node_netstat_TcpExt_TCPOFOMerge 0 # HELP node_netstat_TcpExt_TCPOFOQueue Protocol TcpExt statistic TCPOFOQueue. # TYPE node_netstat_TcpExt_TCPOFOQueue untyped node_netstat_TcpExt_TCPOFOQueue 0 # HELP node_netstat_TcpExt_TCPOrigDataSent Protocol TcpExt statistic TCPOrigDataSent. # TYPE node_netstat_TcpExt_TCPOrigDataSent untyped node_netstat_TcpExt_TCPOrigDataSent 99807 # HELP node_netstat_TcpExt_TCPPartialUndo Protocol TcpExt statistic TCPPartialUndo. # TYPE node_netstat_TcpExt_TCPPartialUndo untyped node_netstat_TcpExt_TCPPartialUndo 0 # HELP node_netstat_TcpExt_TCPPrequeueDropped Protocol TcpExt statistic TCPPrequeueDropped. # TYPE node_netstat_TcpExt_TCPPrequeueDropped untyped node_netstat_TcpExt_TCPPrequeueDropped 0 # HELP node_netstat_TcpExt_TCPPrequeued Protocol TcpExt statistic TCPPrequeued. # TYPE node_netstat_TcpExt_TCPPrequeued untyped node_netstat_TcpExt_TCPPrequeued 68753 # HELP node_netstat_TcpExt_TCPPureAcks Protocol TcpExt statistic TCPPureAcks. # TYPE node_netstat_TcpExt_TCPPureAcks untyped node_netstat_TcpExt_TCPPureAcks 17834 # HELP node_netstat_TcpExt_TCPRcvCoalesce Protocol TcpExt statistic TCPRcvCoalesce. # TYPE node_netstat_TcpExt_TCPRcvCoalesce untyped node_netstat_TcpExt_TCPRcvCoalesce 2 # HELP node_netstat_TcpExt_TCPRcvCollapsed Protocol TcpExt statistic TCPRcvCollapsed. # TYPE node_netstat_TcpExt_TCPRcvCollapsed untyped node_netstat_TcpExt_TCPRcvCollapsed 0 # HELP node_netstat_TcpExt_TCPRenoFailures Protocol TcpExt statistic TCPRenoFailures. # TYPE node_netstat_TcpExt_TCPRenoFailures untyped node_netstat_TcpExt_TCPRenoFailures 0 # HELP node_netstat_TcpExt_TCPRenoRecovery Protocol TcpExt statistic TCPRenoRecovery. # TYPE node_netstat_TcpExt_TCPRenoRecovery untyped node_netstat_TcpExt_TCPRenoRecovery 0 # HELP node_netstat_TcpExt_TCPRenoRecoveryFail Protocol TcpExt statistic TCPRenoRecoveryFail. # TYPE node_netstat_TcpExt_TCPRenoRecoveryFail untyped node_netstat_TcpExt_TCPRenoRecoveryFail 0 # HELP node_netstat_TcpExt_TCPRenoReorder Protocol TcpExt statistic TCPRenoReorder. # TYPE node_netstat_TcpExt_TCPRenoReorder untyped node_netstat_TcpExt_TCPRenoReorder 0 # HELP node_netstat_TcpExt_TCPReqQFullDoCookies Protocol TcpExt statistic TCPReqQFullDoCookies. # TYPE node_netstat_TcpExt_TCPReqQFullDoCookies untyped node_netstat_TcpExt_TCPReqQFullDoCookies 0 # HELP node_netstat_TcpExt_TCPReqQFullDrop Protocol TcpExt statistic TCPReqQFullDrop. # TYPE node_netstat_TcpExt_TCPReqQFullDrop untyped node_netstat_TcpExt_TCPReqQFullDrop 0 # HELP node_netstat_TcpExt_TCPRetransFail Protocol TcpExt statistic TCPRetransFail. # TYPE node_netstat_TcpExt_TCPRetransFail untyped node_netstat_TcpExt_TCPRetransFail 0 # HELP node_netstat_TcpExt_TCPSACKDiscard Protocol TcpExt statistic TCPSACKDiscard. # TYPE node_netstat_TcpExt_TCPSACKDiscard untyped node_netstat_TcpExt_TCPSACKDiscard 0 # HELP node_netstat_TcpExt_TCPSACKReneging Protocol TcpExt statistic TCPSACKReneging. # TYPE node_netstat_TcpExt_TCPSACKReneging untyped node_netstat_TcpExt_TCPSACKReneging 0 # HELP node_netstat_TcpExt_TCPSACKReorder Protocol TcpExt statistic TCPSACKReorder. # TYPE node_netstat_TcpExt_TCPSACKReorder untyped node_netstat_TcpExt_TCPSACKReorder 0 # HELP node_netstat_TcpExt_TCPSYNChallenge Protocol TcpExt statistic TCPSYNChallenge. # TYPE node_netstat_TcpExt_TCPSYNChallenge untyped node_netstat_TcpExt_TCPSYNChallenge 0 # HELP node_netstat_TcpExt_TCPSackFailures Protocol TcpExt statistic TCPSackFailures. # TYPE node_netstat_TcpExt_TCPSackFailures untyped node_netstat_TcpExt_TCPSackFailures 0 # HELP node_netstat_TcpExt_TCPSackMerged Protocol TcpExt statistic TCPSackMerged. # TYPE node_netstat_TcpExt_TCPSackMerged untyped node_netstat_TcpExt_TCPSackMerged 0 # HELP node_netstat_TcpExt_TCPSackRecovery Protocol TcpExt statistic TCPSackRecovery. # TYPE node_netstat_TcpExt_TCPSackRecovery untyped node_netstat_TcpExt_TCPSackRecovery 0 # HELP node_netstat_TcpExt_TCPSackRecoveryFail Protocol TcpExt statistic TCPSackRecoveryFail. # TYPE node_netstat_TcpExt_TCPSackRecoveryFail untyped node_netstat_TcpExt_TCPSackRecoveryFail 0 # HELP node_netstat_TcpExt_TCPSackShiftFallback Protocol TcpExt statistic TCPSackShiftFallback. # TYPE node_netstat_TcpExt_TCPSackShiftFallback untyped node_netstat_TcpExt_TCPSackShiftFallback 0 # HELP node_netstat_TcpExt_TCPSackShifted Protocol TcpExt statistic TCPSackShifted. # TYPE node_netstat_TcpExt_TCPSackShifted untyped node_netstat_TcpExt_TCPSackShifted 0 # HELP node_netstat_TcpExt_TCPSchedulerFailed Protocol TcpExt statistic TCPSchedulerFailed. # TYPE node_netstat_TcpExt_TCPSchedulerFailed untyped node_netstat_TcpExt_TCPSchedulerFailed 0 # HELP node_netstat_TcpExt_TCPSlowStartRetrans Protocol TcpExt statistic TCPSlowStartRetrans. # TYPE node_netstat_TcpExt_TCPSlowStartRetrans untyped node_netstat_TcpExt_TCPSlowStartRetrans 0 # HELP node_netstat_TcpExt_TCPSpuriousRTOs Protocol TcpExt statistic TCPSpuriousRTOs. # TYPE node_netstat_TcpExt_TCPSpuriousRTOs untyped node_netstat_TcpExt_TCPSpuriousRTOs 0 # HELP node_netstat_TcpExt_TCPSpuriousRtxHostQueues Protocol TcpExt statistic TCPSpuriousRtxHostQueues. # TYPE node_netstat_TcpExt_TCPSpuriousRtxHostQueues untyped node_netstat_TcpExt_TCPSpuriousRtxHostQueues 0 # HELP node_netstat_TcpExt_TCPSynRetrans Protocol TcpExt statistic TCPSynRetrans. # TYPE node_netstat_TcpExt_TCPSynRetrans untyped node_netstat_TcpExt_TCPSynRetrans 0 # HELP node_netstat_TcpExt_TCPTSReorder Protocol TcpExt statistic TCPTSReorder. # TYPE node_netstat_TcpExt_TCPTSReorder untyped node_netstat_TcpExt_TCPTSReorder 0 # HELP node_netstat_TcpExt_TCPTimeWaitOverflow Protocol TcpExt statistic TCPTimeWaitOverflow. # TYPE node_netstat_TcpExt_TCPTimeWaitOverflow untyped node_netstat_TcpExt_TCPTimeWaitOverflow 0 # HELP node_netstat_TcpExt_TCPTimeouts Protocol TcpExt statistic TCPTimeouts. # TYPE node_netstat_TcpExt_TCPTimeouts untyped node_netstat_TcpExt_TCPTimeouts 0 # HELP node_netstat_TcpExt_TCPToZeroWindowAdv Protocol TcpExt statistic TCPToZeroWindowAdv. # TYPE node_netstat_TcpExt_TCPToZeroWindowAdv untyped node_netstat_TcpExt_TCPToZeroWindowAdv 0 # HELP node_netstat_TcpExt_TCPWantZeroWindowAdv Protocol TcpExt statistic TCPWantZeroWindowAdv. # TYPE node_netstat_TcpExt_TCPWantZeroWindowAdv untyped node_netstat_TcpExt_TCPWantZeroWindowAdv 0 # HELP node_netstat_TcpExt_TW Protocol TcpExt statistic TW. # TYPE node_netstat_TcpExt_TW untyped node_netstat_TcpExt_TW 35 # HELP node_netstat_TcpExt_TWKilled Protocol TcpExt statistic TWKilled. # TYPE node_netstat_TcpExt_TWKilled untyped node_netstat_TcpExt_TWKilled 0 # HELP node_netstat_TcpExt_TWRecycled Protocol TcpExt statistic TWRecycled. # TYPE node_netstat_TcpExt_TWRecycled untyped node_netstat_TcpExt_TWRecycled 0 # HELP node_netstat_Tcp_ActiveOpens Protocol Tcp statistic ActiveOpens. # TYPE node_netstat_Tcp_ActiveOpens untyped node_netstat_Tcp_ActiveOpens 122 # HELP node_netstat_Tcp_AttemptFails Protocol Tcp statistic AttemptFails. # TYPE node_netstat_Tcp_AttemptFails untyped node_netstat_Tcp_AttemptFails 48 # HELP node_netstat_Tcp_CurrEstab Protocol Tcp statistic CurrEstab. # TYPE node_netstat_Tcp_CurrEstab untyped node_netstat_Tcp_CurrEstab 67 # HELP node_netstat_Tcp_EstabResets Protocol Tcp statistic EstabResets. # TYPE node_netstat_Tcp_EstabResets untyped node_netstat_Tcp_EstabResets 3 # HELP node_netstat_Tcp_InCsumErrors Protocol Tcp statistic InCsumErrors. # TYPE node_netstat_Tcp_InCsumErrors untyped node_netstat_Tcp_InCsumErrors 0 # HELP node_netstat_Tcp_InErrs Protocol Tcp statistic InErrs. # TYPE node_netstat_Tcp_InErrs untyped node_netstat_Tcp_InErrs 0 # HELP node_netstat_Tcp_InSegs Protocol Tcp statistic InSegs. # TYPE node_netstat_Tcp_InSegs untyped node_netstat_Tcp_InSegs 138959 # HELP node_netstat_Tcp_MaxConn Protocol Tcp statistic MaxConn. # TYPE node_netstat_Tcp_MaxConn untyped node_netstat_Tcp_MaxConn -1 # HELP node_netstat_Tcp_OutRsts Protocol Tcp statistic OutRsts. # TYPE node_netstat_Tcp_OutRsts untyped node_netstat_Tcp_OutRsts 51 # HELP node_netstat_Tcp_OutSegs Protocol Tcp statistic OutSegs. # TYPE node_netstat_Tcp_OutSegs untyped node_netstat_Tcp_OutSegs 138800 # HELP node_netstat_Tcp_PassiveOpens Protocol Tcp statistic PassiveOpens. # TYPE node_netstat_Tcp_PassiveOpens untyped node_netstat_Tcp_PassiveOpens 75 # HELP node_netstat_Tcp_RetransSegs Protocol Tcp statistic RetransSegs. # TYPE node_netstat_Tcp_RetransSegs untyped node_netstat_Tcp_RetransSegs 0 # HELP node_netstat_Tcp_RtoAlgorithm Protocol Tcp statistic RtoAlgorithm. # TYPE node_netstat_Tcp_RtoAlgorithm untyped node_netstat_Tcp_RtoAlgorithm 1 # HELP node_netstat_Tcp_RtoMax Protocol Tcp statistic RtoMax. # TYPE node_netstat_Tcp_RtoMax untyped node_netstat_Tcp_RtoMax 120000 # HELP node_netstat_Tcp_RtoMin Protocol Tcp statistic RtoMin. # TYPE node_netstat_Tcp_RtoMin untyped node_netstat_Tcp_RtoMin 200 # HELP node_netstat_UdpLite_InCsumErrors Protocol UdpLite statistic InCsumErrors. # TYPE node_netstat_UdpLite_InCsumErrors untyped node_netstat_UdpLite_InCsumErrors 0 # HELP node_netstat_UdpLite_InDatagrams Protocol UdpLite statistic InDatagrams. # TYPE node_netstat_UdpLite_InDatagrams untyped node_netstat_UdpLite_InDatagrams 0 # HELP node_netstat_UdpLite_InErrors Protocol UdpLite statistic InErrors. # TYPE node_netstat_UdpLite_InErrors untyped node_netstat_UdpLite_InErrors 0 # HELP node_netstat_UdpLite_NoPorts Protocol UdpLite statistic NoPorts. # TYPE node_netstat_UdpLite_NoPorts untyped node_netstat_UdpLite_NoPorts 0 # HELP node_netstat_UdpLite_OutDatagrams Protocol UdpLite statistic OutDatagrams. # TYPE node_netstat_UdpLite_OutDatagrams untyped node_netstat_UdpLite_OutDatagrams 0 # HELP node_netstat_UdpLite_RcvbufErrors Protocol UdpLite statistic RcvbufErrors. # TYPE node_netstat_UdpLite_RcvbufErrors untyped node_netstat_UdpLite_RcvbufErrors 0 # HELP node_netstat_UdpLite_SndbufErrors Protocol UdpLite statistic SndbufErrors. # TYPE node_netstat_UdpLite_SndbufErrors untyped node_netstat_UdpLite_SndbufErrors 0 # HELP node_netstat_Udp_InCsumErrors Protocol Udp statistic InCsumErrors. # TYPE node_netstat_Udp_InCsumErrors untyped node_netstat_Udp_InCsumErrors 0 # HELP node_netstat_Udp_InDatagrams Protocol Udp statistic InDatagrams. # TYPE node_netstat_Udp_InDatagrams untyped node_netstat_Udp_InDatagrams 347 # HELP node_netstat_Udp_InErrors Protocol Udp statistic InErrors. # TYPE node_netstat_Udp_InErrors untyped node_netstat_Udp_InErrors 0 # HELP node_netstat_Udp_NoPorts Protocol Udp statistic NoPorts. # TYPE node_netstat_Udp_NoPorts untyped node_netstat_Udp_NoPorts 32 # HELP node_netstat_Udp_OutDatagrams Protocol Udp statistic OutDatagrams. # TYPE node_netstat_Udp_OutDatagrams untyped node_netstat_Udp_OutDatagrams 388 # HELP node_netstat_Udp_RcvbufErrors Protocol Udp statistic RcvbufErrors. # TYPE node_netstat_Udp_RcvbufErrors untyped node_netstat_Udp_RcvbufErrors 0 # HELP node_netstat_Udp_SndbufErrors Protocol Udp statistic SndbufErrors. # TYPE node_netstat_Udp_SndbufErrors untyped node_netstat_Udp_SndbufErrors 0 # HELP node_network_receive_bytes Network device statistic receive_bytes. # TYPE node_network_receive_bytes gauge node_network_receive_bytes{device="docker0"} 0 node_network_receive_bytes{device="enp0s3"} 36421 node_network_receive_bytes{device="enp0s8"} 80451 node_network_receive_bytes{device="lo"} 3.4463599e+07 # HELP node_network_receive_compressed Network device statistic receive_compressed. # TYPE node_network_receive_compressed gauge node_network_receive_compressed{device="docker0"} 0 node_network_receive_compressed{device="enp0s3"} 0 node_network_receive_compressed{device="enp0s8"} 0 node_network_receive_compressed{device="lo"} 0 # HELP node_network_receive_drop Network device statistic receive_drop. # TYPE node_network_receive_drop gauge node_network_receive_drop{device="docker0"} 0 node_network_receive_drop{device="enp0s3"} 0 node_network_receive_drop{device="enp0s8"} 0 node_network_receive_drop{device="lo"} 0 # HELP node_network_receive_errs Network device statistic receive_errs. # TYPE node_network_receive_errs gauge node_network_receive_errs{device="docker0"} 0 node_network_receive_errs{device="enp0s3"} 0 node_network_receive_errs{device="enp0s8"} 0 node_network_receive_errs{device="lo"} 0 # HELP node_network_receive_fifo Network device statistic receive_fifo. # TYPE node_network_receive_fifo gauge node_network_receive_fifo{device="docker0"} 0 node_network_receive_fifo{device="enp0s3"} 0 node_network_receive_fifo{device="enp0s8"} 0 node_network_receive_fifo{device="lo"} 0 # HELP node_network_receive_frame Network device statistic receive_frame. # TYPE node_network_receive_frame gauge node_network_receive_frame{device="docker0"} 0 node_network_receive_frame{device="enp0s3"} 0 node_network_receive_frame{device="enp0s8"} 0 node_network_receive_frame{device="lo"} 0 # HELP node_network_receive_multicast Network device statistic receive_multicast. # TYPE node_network_receive_multicast gauge node_network_receive_multicast{device="docker0"} 0 node_network_receive_multicast{device="enp0s3"} 0 node_network_receive_multicast{device="enp0s8"} 0 node_network_receive_multicast{device="lo"} 0 # HELP node_network_receive_packets Network device statistic receive_packets. # TYPE node_network_receive_packets gauge node_network_receive_packets{device="docker0"} 0 node_network_receive_packets{device="enp0s3"} 421 node_network_receive_packets{device="enp0s8"} 997 node_network_receive_packets{device="lo"} 138313 # HELP node_network_transmit_bytes Network device statistic transmit_bytes. # TYPE node_network_transmit_bytes gauge node_network_transmit_bytes{device="docker0"} 0 node_network_transmit_bytes{device="enp0s3"} 37074 node_network_transmit_bytes{device="enp0s8"} 330717 node_network_transmit_bytes{device="lo"} 3.4463599e+07 # HELP node_network_transmit_compressed Network device statistic transmit_compressed. # TYPE node_network_transmit_compressed gauge node_network_transmit_compressed{device="docker0"} 0 node_network_transmit_compressed{device="enp0s3"} 0 node_network_transmit_compressed{device="enp0s8"} 0 node_network_transmit_compressed{device="lo"} 0 # HELP node_network_transmit_drop Network device statistic transmit_drop. # TYPE node_network_transmit_drop gauge node_network_transmit_drop{device="docker0"} 0 node_network_transmit_drop{device="enp0s3"} 0 node_network_transmit_drop{device="enp0s8"} 0 node_network_transmit_drop{device="lo"} 0 # HELP node_network_transmit_errs Network device statistic transmit_errs. # TYPE node_network_transmit_errs gauge node_network_transmit_errs{device="docker0"} 0 node_network_transmit_errs{device="enp0s3"} 0 node_network_transmit_errs{device="enp0s8"} 0 node_network_transmit_errs{device="lo"} 0 # HELP node_network_transmit_fifo Network device statistic transmit_fifo. # TYPE node_network_transmit_fifo gauge node_network_transmit_fifo{device="docker0"} 0 node_network_transmit_fifo{device="enp0s3"} 0 node_network_transmit_fifo{device="enp0s8"} 0 node_network_transmit_fifo{device="lo"} 0 # HELP node_network_transmit_frame Network device statistic transmit_frame. # TYPE node_network_transmit_frame gauge node_network_transmit_frame{device="docker0"} 0 node_network_transmit_frame{device="enp0s3"} 0 node_network_transmit_frame{device="enp0s8"} 0 node_network_transmit_frame{device="lo"} 0 # HELP node_network_transmit_multicast Network device statistic transmit_multicast. # TYPE node_network_transmit_multicast gauge node_network_transmit_multicast{device="docker0"} 0 node_network_transmit_multicast{device="enp0s3"} 0 node_network_transmit_multicast{device="enp0s8"} 0 node_network_transmit_multicast{device="lo"} 0 # HELP node_network_transmit_packets Network device statistic transmit_packets. # TYPE node_network_transmit_packets gauge node_network_transmit_packets{device="docker0"} 0 node_network_transmit_packets{device="enp0s3"} 438 node_network_transmit_packets{device="enp0s8"} 627 node_network_transmit_packets{device="lo"} 138313 # HELP node_procs_blocked Number of processes blocked waiting for I/O to complete. # TYPE node_procs_blocked gauge node_procs_blocked 0 # HELP node_procs_running Number of processes in runnable state. # TYPE node_procs_running gauge node_procs_running 2 # HELP node_time System time in seconds since epoch (1970). # TYPE node_time counter node_time 1.48891092e+09 # HELP node_uname_info Labeled system information as provided by the uname system call. # TYPE node_uname_info gauge node_uname_info{domainname="(none)",machine="x86_64",nodename="centos7",release="3.10.0-514.10.2.el7.x86_64",sysname="Linux",version="#1 SMP Fri Mar 3 00:04:05 UTC 2017"} 1 # HELP node_vmstat_allocstall /proc/vmstat information field allocstall. # TYPE node_vmstat_allocstall untyped node_vmstat_allocstall 0 # HELP node_vmstat_balloon_deflate /proc/vmstat information field balloon_deflate. # TYPE node_vmstat_balloon_deflate untyped node_vmstat_balloon_deflate 0 # HELP node_vmstat_balloon_inflate /proc/vmstat information field balloon_inflate. # TYPE node_vmstat_balloon_inflate untyped node_vmstat_balloon_inflate 0 # HELP node_vmstat_balloon_migrate /proc/vmstat information field balloon_migrate. # TYPE node_vmstat_balloon_migrate untyped node_vmstat_balloon_migrate 0 # HELP node_vmstat_compact_fail /proc/vmstat information field compact_fail. # TYPE node_vmstat_compact_fail untyped node_vmstat_compact_fail 0 # HELP node_vmstat_compact_free_scanned /proc/vmstat information field compact_free_scanned. # TYPE node_vmstat_compact_free_scanned untyped node_vmstat_compact_free_scanned 0 # HELP node_vmstat_compact_isolated /proc/vmstat information field compact_isolated. # TYPE node_vmstat_compact_isolated untyped node_vmstat_compact_isolated 0 # HELP node_vmstat_compact_migrate_scanned /proc/vmstat information field compact_migrate_scanned. # TYPE node_vmstat_compact_migrate_scanned untyped node_vmstat_compact_migrate_scanned 0 # HELP node_vmstat_compact_stall /proc/vmstat information field compact_stall. # TYPE node_vmstat_compact_stall untyped node_vmstat_compact_stall 0 # HELP node_vmstat_compact_success /proc/vmstat information field compact_success. # TYPE node_vmstat_compact_success untyped node_vmstat_compact_success 0 # HELP node_vmstat_drop_pagecache /proc/vmstat information field drop_pagecache. # TYPE node_vmstat_drop_pagecache untyped node_vmstat_drop_pagecache 0 # HELP node_vmstat_drop_slab /proc/vmstat information field drop_slab. # TYPE node_vmstat_drop_slab untyped node_vmstat_drop_slab 0 # HELP node_vmstat_htlb_buddy_alloc_fail /proc/vmstat information field htlb_buddy_alloc_fail. # TYPE node_vmstat_htlb_buddy_alloc_fail untyped node_vmstat_htlb_buddy_alloc_fail 0 # HELP node_vmstat_htlb_buddy_alloc_success /proc/vmstat information field htlb_buddy_alloc_success. # TYPE node_vmstat_htlb_buddy_alloc_success untyped node_vmstat_htlb_buddy_alloc_success 0 # HELP node_vmstat_kswapd_high_wmark_hit_quickly /proc/vmstat information field kswapd_high_wmark_hit_quickly. # TYPE node_vmstat_kswapd_high_wmark_hit_quickly untyped node_vmstat_kswapd_high_wmark_hit_quickly 0 # HELP node_vmstat_kswapd_inodesteal /proc/vmstat information field kswapd_inodesteal. # TYPE node_vmstat_kswapd_inodesteal untyped node_vmstat_kswapd_inodesteal 0 # HELP node_vmstat_kswapd_low_wmark_hit_quickly /proc/vmstat information field kswapd_low_wmark_hit_quickly. # TYPE node_vmstat_kswapd_low_wmark_hit_quickly untyped node_vmstat_kswapd_low_wmark_hit_quickly 0 # HELP node_vmstat_nr_active_anon /proc/vmstat information field nr_active_anon. # TYPE node_vmstat_nr_active_anon untyped node_vmstat_nr_active_anon 88528 # HELP node_vmstat_nr_active_file /proc/vmstat information field nr_active_file. # TYPE node_vmstat_nr_active_file untyped node_vmstat_nr_active_file 28848 # HELP node_vmstat_nr_alloc_batch /proc/vmstat information field nr_alloc_batch. # TYPE node_vmstat_nr_alloc_batch untyped node_vmstat_nr_alloc_batch 2642 # HELP node_vmstat_nr_anon_pages /proc/vmstat information field nr_anon_pages. # TYPE node_vmstat_nr_anon_pages untyped node_vmstat_nr_anon_pages 85414 # HELP node_vmstat_nr_anon_transparent_hugepages /proc/vmstat information field nr_anon_transparent_hugepages. # TYPE node_vmstat_nr_anon_transparent_hugepages untyped node_vmstat_nr_anon_transparent_hugepages 6 # HELP node_vmstat_nr_bounce /proc/vmstat information field nr_bounce. # TYPE node_vmstat_nr_bounce untyped node_vmstat_nr_bounce 0 # HELP node_vmstat_nr_dirtied /proc/vmstat information field nr_dirtied. # TYPE node_vmstat_nr_dirtied untyped node_vmstat_nr_dirtied 29852 # HELP node_vmstat_nr_dirty /proc/vmstat information field nr_dirty. # TYPE node_vmstat_nr_dirty untyped node_vmstat_nr_dirty 17 # HELP node_vmstat_nr_dirty_background_threshold /proc/vmstat information field nr_dirty_background_threshold. # TYPE node_vmstat_nr_dirty_background_threshold untyped node_vmstat_nr_dirty_background_threshold 39900 # HELP node_vmstat_nr_dirty_threshold /proc/vmstat information field nr_dirty_threshold. # TYPE node_vmstat_nr_dirty_threshold untyped node_vmstat_nr_dirty_threshold 119701 # HELP node_vmstat_nr_file_pages /proc/vmstat information field nr_file_pages. # TYPE node_vmstat_nr_file_pages untyped node_vmstat_nr_file_pages 76687 # HELP node_vmstat_nr_free_cma /proc/vmstat information field nr_free_cma. # TYPE node_vmstat_nr_free_cma untyped node_vmstat_nr_free_cma 0 # HELP node_vmstat_nr_free_pages /proc/vmstat information field nr_free_pages. # TYPE node_vmstat_nr_free_pages untyped node_vmstat_nr_free_pages 769929 # HELP node_vmstat_nr_inactive_anon /proc/vmstat information field nr_inactive_anon. # TYPE node_vmstat_nr_inactive_anon untyped node_vmstat_nr_inactive_anon 2091 # HELP node_vmstat_nr_inactive_file /proc/vmstat information field nr_inactive_file. # TYPE node_vmstat_nr_inactive_file untyped node_vmstat_nr_inactive_file 45688 # HELP node_vmstat_nr_isolated_anon /proc/vmstat information field nr_isolated_anon. # TYPE node_vmstat_nr_isolated_anon untyped node_vmstat_nr_isolated_anon 0 # HELP node_vmstat_nr_isolated_file /proc/vmstat information field nr_isolated_file. # TYPE node_vmstat_nr_isolated_file untyped node_vmstat_nr_isolated_file 0 # HELP node_vmstat_nr_kernel_stack /proc/vmstat information field nr_kernel_stack. # TYPE node_vmstat_nr_kernel_stack untyped node_vmstat_nr_kernel_stack 707 # HELP node_vmstat_nr_mapped /proc/vmstat information field nr_mapped. # TYPE node_vmstat_nr_mapped untyped node_vmstat_nr_mapped 16487 # HELP node_vmstat_nr_mlock /proc/vmstat information field nr_mlock. # TYPE node_vmstat_nr_mlock untyped node_vmstat_nr_mlock 0 # HELP node_vmstat_nr_page_table_pages /proc/vmstat information field nr_page_table_pages. # TYPE node_vmstat_nr_page_table_pages untyped node_vmstat_nr_page_table_pages 2223 # HELP node_vmstat_nr_shmem /proc/vmstat information field nr_shmem. # TYPE node_vmstat_nr_shmem untyped node_vmstat_nr_shmem 2151 # HELP node_vmstat_nr_slab_reclaimable /proc/vmstat information field nr_slab_reclaimable. # TYPE node_vmstat_nr_slab_reclaimable untyped node_vmstat_nr_slab_reclaimable 10898 # HELP node_vmstat_nr_slab_unreclaimable /proc/vmstat information field nr_slab_unreclaimable. # TYPE node_vmstat_nr_slab_unreclaimable untyped node_vmstat_nr_slab_unreclaimable 7827 # HELP node_vmstat_nr_unevictable /proc/vmstat information field nr_unevictable. # TYPE node_vmstat_nr_unevictable untyped node_vmstat_nr_unevictable 0 # HELP node_vmstat_nr_unstable /proc/vmstat information field nr_unstable. # TYPE node_vmstat_nr_unstable untyped node_vmstat_nr_unstable 0 # HELP node_vmstat_nr_vmscan_immediate_reclaim /proc/vmstat information field nr_vmscan_immediate_reclaim. # TYPE node_vmstat_nr_vmscan_immediate_reclaim untyped node_vmstat_nr_vmscan_immediate_reclaim 0 # HELP node_vmstat_nr_vmscan_write /proc/vmstat information field nr_vmscan_write. # TYPE node_vmstat_nr_vmscan_write untyped node_vmstat_nr_vmscan_write 0 # HELP node_vmstat_nr_writeback /proc/vmstat information field nr_writeback. # TYPE node_vmstat_nr_writeback untyped node_vmstat_nr_writeback 0 # HELP node_vmstat_nr_writeback_temp /proc/vmstat information field nr_writeback_temp. # TYPE node_vmstat_nr_writeback_temp untyped node_vmstat_nr_writeback_temp 0 # HELP node_vmstat_nr_written /proc/vmstat information field nr_written. # TYPE node_vmstat_nr_written untyped node_vmstat_nr_written 20344 # HELP node_vmstat_numa_foreign /proc/vmstat information field numa_foreign. # TYPE node_vmstat_numa_foreign untyped node_vmstat_numa_foreign 0 # HELP node_vmstat_numa_hint_faults /proc/vmstat information field numa_hint_faults. # TYPE node_vmstat_numa_hint_faults untyped node_vmstat_numa_hint_faults 0 # HELP node_vmstat_numa_hint_faults_local /proc/vmstat information field numa_hint_faults_local. # TYPE node_vmstat_numa_hint_faults_local untyped node_vmstat_numa_hint_faults_local 0 # HELP node_vmstat_numa_hit /proc/vmstat information field numa_hit. # TYPE node_vmstat_numa_hit untyped node_vmstat_numa_hit 824961 # HELP node_vmstat_numa_huge_pte_updates /proc/vmstat information field numa_huge_pte_updates. # TYPE node_vmstat_numa_huge_pte_updates untyped node_vmstat_numa_huge_pte_updates 0 # HELP node_vmstat_numa_interleave /proc/vmstat information field numa_interleave. # TYPE node_vmstat_numa_interleave untyped node_vmstat_numa_interleave 14563 # HELP node_vmstat_numa_local /proc/vmstat information field numa_local. # TYPE node_vmstat_numa_local untyped node_vmstat_numa_local 824961 # HELP node_vmstat_numa_miss /proc/vmstat information field numa_miss. # TYPE node_vmstat_numa_miss untyped node_vmstat_numa_miss 0 # HELP node_vmstat_numa_other /proc/vmstat information field numa_other. # TYPE node_vmstat_numa_other untyped node_vmstat_numa_other 0 # HELP node_vmstat_numa_pages_migrated /proc/vmstat information field numa_pages_migrated. # TYPE node_vmstat_numa_pages_migrated untyped node_vmstat_numa_pages_migrated 0 # HELP node_vmstat_numa_pte_updates /proc/vmstat information field numa_pte_updates. # TYPE node_vmstat_numa_pte_updates untyped node_vmstat_numa_pte_updates 0 # HELP node_vmstat_pageoutrun /proc/vmstat information field pageoutrun. # TYPE node_vmstat_pageoutrun untyped node_vmstat_pageoutrun 1 # HELP node_vmstat_pgactivate /proc/vmstat information field pgactivate. # TYPE node_vmstat_pgactivate untyped node_vmstat_pgactivate 30452 # HELP node_vmstat_pgalloc_dma /proc/vmstat information field pgalloc_dma. # TYPE node_vmstat_pgalloc_dma untyped node_vmstat_pgalloc_dma 487 # HELP node_vmstat_pgalloc_dma32 /proc/vmstat information field pgalloc_dma32. # TYPE node_vmstat_pgalloc_dma32 untyped node_vmstat_pgalloc_dma32 750526 # HELP node_vmstat_pgalloc_movable /proc/vmstat information field pgalloc_movable. # TYPE node_vmstat_pgalloc_movable untyped node_vmstat_pgalloc_movable 0 # HELP node_vmstat_pgalloc_normal /proc/vmstat information field pgalloc_normal. # TYPE node_vmstat_pgalloc_normal untyped node_vmstat_pgalloc_normal 124283 # HELP node_vmstat_pgdeactivate /proc/vmstat information field pgdeactivate. # TYPE node_vmstat_pgdeactivate untyped node_vmstat_pgdeactivate 0 # HELP node_vmstat_pgfault /proc/vmstat information field pgfault. # TYPE node_vmstat_pgfault untyped node_vmstat_pgfault 1.37842e+06 # HELP node_vmstat_pgfree /proc/vmstat information field pgfree. # TYPE node_vmstat_pgfree untyped node_vmstat_pgfree 1.645789e+06 # HELP node_vmstat_pginodesteal /proc/vmstat information field pginodesteal. # TYPE node_vmstat_pginodesteal untyped node_vmstat_pginodesteal 0 # HELP node_vmstat_pgmajfault /proc/vmstat information field pgmajfault. # TYPE node_vmstat_pgmajfault untyped node_vmstat_pgmajfault 4286 # HELP node_vmstat_pgmigrate_fail /proc/vmstat information field pgmigrate_fail. # TYPE node_vmstat_pgmigrate_fail untyped node_vmstat_pgmigrate_fail 0 # HELP node_vmstat_pgmigrate_success /proc/vmstat information field pgmigrate_success. # TYPE node_vmstat_pgmigrate_success untyped node_vmstat_pgmigrate_success 0 # HELP node_vmstat_pgpgin /proc/vmstat information field pgpgin. # TYPE node_vmstat_pgpgin untyped node_vmstat_pgpgin 289776 # HELP node_vmstat_pgpgout /proc/vmstat information field pgpgout. # TYPE node_vmstat_pgpgout untyped node_vmstat_pgpgout 139684 # HELP node_vmstat_pgrefill_dma /proc/vmstat information field pgrefill_dma. # TYPE node_vmstat_pgrefill_dma untyped node_vmstat_pgrefill_dma 0 # HELP node_vmstat_pgrefill_dma32 /proc/vmstat information field pgrefill_dma32. # TYPE node_vmstat_pgrefill_dma32 untyped node_vmstat_pgrefill_dma32 0 # HELP node_vmstat_pgrefill_movable /proc/vmstat information field pgrefill_movable. # TYPE node_vmstat_pgrefill_movable untyped node_vmstat_pgrefill_movable 0 # HELP node_vmstat_pgrefill_normal /proc/vmstat information field pgrefill_normal. # TYPE node_vmstat_pgrefill_normal untyped node_vmstat_pgrefill_normal 0 # HELP node_vmstat_pgrotated /proc/vmstat information field pgrotated. # TYPE node_vmstat_pgrotated untyped node_vmstat_pgrotated 0 # HELP node_vmstat_pgscan_direct_dma /proc/vmstat information field pgscan_direct_dma. # TYPE node_vmstat_pgscan_direct_dma untyped node_vmstat_pgscan_direct_dma 0 # HELP node_vmstat_pgscan_direct_dma32 /proc/vmstat information field pgscan_direct_dma32. # TYPE node_vmstat_pgscan_direct_dma32 untyped node_vmstat_pgscan_direct_dma32 0 # HELP node_vmstat_pgscan_direct_movable /proc/vmstat information field pgscan_direct_movable. # TYPE node_vmstat_pgscan_direct_movable untyped node_vmstat_pgscan_direct_movable 0 # HELP node_vmstat_pgscan_direct_normal /proc/vmstat information field pgscan_direct_normal. # TYPE node_vmstat_pgscan_direct_normal untyped node_vmstat_pgscan_direct_normal 0 # HELP node_vmstat_pgscan_direct_throttle /proc/vmstat information field pgscan_direct_throttle. # TYPE node_vmstat_pgscan_direct_throttle untyped node_vmstat_pgscan_direct_throttle 0 # HELP node_vmstat_pgscan_kswapd_dma /proc/vmstat information field pgscan_kswapd_dma. # TYPE node_vmstat_pgscan_kswapd_dma untyped node_vmstat_pgscan_kswapd_dma 0 # HELP node_vmstat_pgscan_kswapd_dma32 /proc/vmstat information field pgscan_kswapd_dma32. # TYPE node_vmstat_pgscan_kswapd_dma32 untyped node_vmstat_pgscan_kswapd_dma32 0 # HELP node_vmstat_pgscan_kswapd_movable /proc/vmstat information field pgscan_kswapd_movable. # TYPE node_vmstat_pgscan_kswapd_movable untyped node_vmstat_pgscan_kswapd_movable 0 # HELP node_vmstat_pgscan_kswapd_normal /proc/vmstat information field pgscan_kswapd_normal. # TYPE node_vmstat_pgscan_kswapd_normal untyped node_vmstat_pgscan_kswapd_normal 0 # HELP node_vmstat_pgsteal_direct_dma /proc/vmstat information field pgsteal_direct_dma. # TYPE node_vmstat_pgsteal_direct_dma untyped node_vmstat_pgsteal_direct_dma 0 # HELP node_vmstat_pgsteal_direct_dma32 /proc/vmstat information field pgsteal_direct_dma32. # TYPE node_vmstat_pgsteal_direct_dma32 untyped node_vmstat_pgsteal_direct_dma32 0 # HELP node_vmstat_pgsteal_direct_movable /proc/vmstat information field pgsteal_direct_movable. # TYPE node_vmstat_pgsteal_direct_movable untyped node_vmstat_pgsteal_direct_movable 0 # HELP node_vmstat_pgsteal_direct_normal /proc/vmstat information field pgsteal_direct_normal. # TYPE node_vmstat_pgsteal_direct_normal untyped node_vmstat_pgsteal_direct_normal 0 # HELP node_vmstat_pgsteal_kswapd_dma /proc/vmstat information field pgsteal_kswapd_dma. # TYPE node_vmstat_pgsteal_kswapd_dma untyped node_vmstat_pgsteal_kswapd_dma 0 # HELP node_vmstat_pgsteal_kswapd_dma32 /proc/vmstat information field pgsteal_kswapd_dma32. # TYPE node_vmstat_pgsteal_kswapd_dma32 untyped node_vmstat_pgsteal_kswapd_dma32 0 # HELP node_vmstat_pgsteal_kswapd_movable /proc/vmstat information field pgsteal_kswapd_movable. # TYPE node_vmstat_pgsteal_kswapd_movable untyped node_vmstat_pgsteal_kswapd_movable 0 # HELP node_vmstat_pgsteal_kswapd_normal /proc/vmstat information field pgsteal_kswapd_normal. # TYPE node_vmstat_pgsteal_kswapd_normal untyped node_vmstat_pgsteal_kswapd_normal 0 # HELP node_vmstat_pswpin /proc/vmstat information field pswpin. # TYPE node_vmstat_pswpin untyped node_vmstat_pswpin 0 # HELP node_vmstat_pswpout /proc/vmstat information field pswpout. # TYPE node_vmstat_pswpout untyped node_vmstat_pswpout 0 # HELP node_vmstat_slabs_scanned /proc/vmstat information field slabs_scanned. # TYPE node_vmstat_slabs_scanned untyped node_vmstat_slabs_scanned 0 # HELP node_vmstat_thp_collapse_alloc /proc/vmstat information field thp_collapse_alloc. # TYPE node_vmstat_thp_collapse_alloc untyped node_vmstat_thp_collapse_alloc 0 # HELP node_vmstat_thp_collapse_alloc_failed /proc/vmstat information field thp_collapse_alloc_failed. # TYPE node_vmstat_thp_collapse_alloc_failed untyped node_vmstat_thp_collapse_alloc_failed 0 # HELP node_vmstat_thp_fault_alloc /proc/vmstat information field thp_fault_alloc. # TYPE node_vmstat_thp_fault_alloc untyped node_vmstat_thp_fault_alloc 53 # HELP node_vmstat_thp_fault_fallback /proc/vmstat information field thp_fault_fallback. # TYPE node_vmstat_thp_fault_fallback untyped node_vmstat_thp_fault_fallback 0 # HELP node_vmstat_thp_split /proc/vmstat information field thp_split. # TYPE node_vmstat_thp_split untyped node_vmstat_thp_split 1 # HELP node_vmstat_thp_zero_page_alloc /proc/vmstat information field thp_zero_page_alloc. # TYPE node_vmstat_thp_zero_page_alloc untyped node_vmstat_thp_zero_page_alloc 0 # HELP node_vmstat_thp_zero_page_alloc_failed /proc/vmstat information field thp_zero_page_alloc_failed. # TYPE node_vmstat_thp_zero_page_alloc_failed untyped node_vmstat_thp_zero_page_alloc_failed 0 # HELP node_vmstat_unevictable_pgs_cleared /proc/vmstat information field unevictable_pgs_cleared. # TYPE node_vmstat_unevictable_pgs_cleared untyped node_vmstat_unevictable_pgs_cleared 0 # HELP node_vmstat_unevictable_pgs_culled /proc/vmstat information field unevictable_pgs_culled. # TYPE node_vmstat_unevictable_pgs_culled untyped node_vmstat_unevictable_pgs_culled 8348 # HELP node_vmstat_unevictable_pgs_mlocked /proc/vmstat information field unevictable_pgs_mlocked. # TYPE node_vmstat_unevictable_pgs_mlocked untyped node_vmstat_unevictable_pgs_mlocked 11226 # HELP node_vmstat_unevictable_pgs_munlocked /proc/vmstat information field unevictable_pgs_munlocked. # TYPE node_vmstat_unevictable_pgs_munlocked untyped node_vmstat_unevictable_pgs_munlocked 9966 # HELP node_vmstat_unevictable_pgs_rescued /proc/vmstat information field unevictable_pgs_rescued. # TYPE node_vmstat_unevictable_pgs_rescued untyped node_vmstat_unevictable_pgs_rescued 7446 # HELP node_vmstat_unevictable_pgs_scanned /proc/vmstat information field unevictable_pgs_scanned. # TYPE node_vmstat_unevictable_pgs_scanned untyped node_vmstat_unevictable_pgs_scanned 0 # HELP node_vmstat_unevictable_pgs_stranded /proc/vmstat information field unevictable_pgs_stranded. # TYPE node_vmstat_unevictable_pgs_stranded untyped node_vmstat_unevictable_pgs_stranded 0 # HELP node_vmstat_workingset_activate /proc/vmstat information field workingset_activate. # TYPE node_vmstat_workingset_activate untyped node_vmstat_workingset_activate 0 # HELP node_vmstat_workingset_nodereclaim /proc/vmstat information field workingset_nodereclaim. # TYPE node_vmstat_workingset_nodereclaim untyped node_vmstat_workingset_nodereclaim 0 # HELP node_vmstat_workingset_refault /proc/vmstat information field workingset_refault. # TYPE node_vmstat_workingset_refault untyped node_vmstat_workingset_refault 0 # HELP node_vmstat_zone_reclaim_failed /proc/vmstat information field zone_reclaim_failed. # TYPE node_vmstat_zone_reclaim_failed untyped node_vmstat_zone_reclaim_failed 0

percona/mongodb_exporter (mongodb:metrics)

Behind the scenes of the “mongodb:metric” Percona Monitoring and Management service, percona/mongodb_exporter is the Prometheus exporter that provides detailed database metrics for graphing on the Percona Monitoring and Management server. percona/mongodb_exporter is a Percona fork of the project dcu/mongodb_exporter, with some valuable additional metrics for MongoDB sharding, storage engines, etc.

The mongodb_exporter process is designed to automatically detect the MongoDB node type, storage engine, etc., without any special configuration aside from the connection string.

As of Percona Monitoring and Management 1.1.1, here are the number of metrics collected from a single replication-enabled mongodb instance on a single ‘pull’ from Prometheus:

  • Percona Server for MongoDB 3.2 w/WiredTiger: 173 metrics per “pull”
  • Percona Server for MongoDB 3.2 w/RocksDB (with 3 x levels): 239 metrics per “pul”
  • Percona Server for MongoDB 3.2 w/MMAPv1: 172 metrics per “pul”‘

Note: each additional replica set member adds an additional seven metrics to the list of metrics.

On the sharding side of things, a “mongos” process within a cluster with one shard reports 58 metrics, with one extra metric added for each additional cluster shard and 3-4 extra metrics added for each additional “mongos” instance.

Below is a full example of a single “pull” of metrics from one RocksDB instance. Prometheus exporters provide metrics at the HTTP URL: “/metrics”. This is the exact same payload Prometheus would poll from the exporter:

$ curl -sk https://192.168.99.10:42003/metrics | grep mongodb # HELP mongodb_mongod_asserts_total The asserts document reports the number of asserts on the database. While assert errors are typically uncommon, if there are non-zero values for the asserts, you should check the log file for the mongod process for more information. In many cases these errors are trivial, but are worth investigating. # TYPE mongodb_mongod_asserts_total counter mongodb_mongod_asserts_total{type="msg"} 0 mongodb_mongod_asserts_total{type="regular"} 0 mongodb_mongod_asserts_total{type="rollovers"} 0 mongodb_mongod_asserts_total{type="user"} 1 mongodb_mongod_asserts_total{type="warning"} 0 # HELP mongodb_mongod_connections The connections sub document data regarding the current status of incoming connections and availability of the database server. Use these values to assess the current load and capacity requirements of the server # TYPE mongodb_mongod_connections gauge mongodb_mongod_connections{state="available"} 811 mongodb_mongod_connections{state="current"} 8 # HELP mongodb_mongod_connections_metrics_created_total totalCreated provides a count of all incoming connections created to the server. This number includes connections that have since closed # TYPE mongodb_mongod_connections_metrics_created_total counter mongodb_mongod_connections_metrics_created_total 1977 # HELP mongodb_mongod_extra_info_heap_usage_bytes The heap_usage_bytes field is only available on Unix/Linux systems, and reports the total size in bytes of heap space used by the database process # TYPE mongodb_mongod_extra_info_heap_usage_bytes gauge mongodb_mongod_extra_info_heap_usage_bytes 7.77654e+07 # HELP mongodb_mongod_extra_info_page_faults_total The page_faults Reports the total number of page faults that require disk operations. Page faults refer to operations that require the database server to access data which isn’t available in active memory. The page_faults counter may increase dramatically during moments of poor performance and may correlate with limited memory environments and larger data sets. Limited and sporadic page faults do not necessarily indicate an issue # TYPE mongodb_mongod_extra_info_page_faults_total gauge mongodb_mongod_extra_info_page_faults_total 47 # HELP mongodb_mongod_global_lock_client The activeClients data structure provides more granular information about the number of connected clients and the operation types (e.g. read or write) performed by these clients # TYPE mongodb_mongod_global_lock_client gauge mongodb_mongod_global_lock_client{type="reader"} 0 mongodb_mongod_global_lock_client{type="writer"} 0 # HELP mongodb_mongod_global_lock_current_queue The currentQueue data structure value provides more granular information concerning the number of operations queued because of a lock # TYPE mongodb_mongod_global_lock_current_queue gauge mongodb_mongod_global_lock_current_queue{type="reader"} 0 mongodb_mongod_global_lock_current_queue{type="writer"} 0 # HELP mongodb_mongod_global_lock_ratio The value of ratio displays the relationship between lockTime and totalTime. Low values indicate that operations have held the globalLock frequently for shorter periods of time. High values indicate that operations have held globalLock infrequently for longer periods of time # TYPE mongodb_mongod_global_lock_ratio gauge mongodb_mongod_global_lock_ratio 0 # HELP mongodb_mongod_global_lock_total The value of totalTime represents the time, in microseconds, since the database last started and creation of the globalLock. This is roughly equivalent to total server uptime # TYPE mongodb_mongod_global_lock_total counter mongodb_mongod_global_lock_total 0 # HELP mongodb_mongod_instance_local_time The localTime value is the current time, according to the server, in UTC specified in an ISODate format. # TYPE mongodb_mongod_instance_local_time counter mongodb_mongod_instance_local_time 1.488834066e+09 # HELP mongodb_mongod_instance_uptime_estimate_seconds uptimeEstimate provides the uptime as calculated from MongoDB's internal course-grained time keeping system. # TYPE mongodb_mongod_instance_uptime_estimate_seconds counter mongodb_mongod_instance_uptime_estimate_seconds 15881 # HELP mongodb_mongod_instance_uptime_seconds The value of the uptime field corresponds to the number of seconds that the mongos or mongod process has been active. # TYPE mongodb_mongod_instance_uptime_seconds counter mongodb_mongod_instance_uptime_seconds 15881 # HELP mongodb_mongod_locks_time_acquiring_global_microseconds_total amount of time in microseconds that any database has spent waiting for the global lock # TYPE mongodb_mongod_locks_time_acquiring_global_microseconds_total counter mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Collection",type="read"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Collection",type="write"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Database",type="read"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Database",type="write"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Global",type="read"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Global",type="write"} 5005 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Metadata",type="read"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Metadata",type="write"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="oplog",type="read"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="oplog",type="write"} 0 # HELP mongodb_mongod_locks_time_locked_global_microseconds_total amount of time in microseconds that any database has held the global lock # TYPE mongodb_mongod_locks_time_locked_global_microseconds_total counter mongodb_mongod_locks_time_locked_global_microseconds_total{database="Collection",type="read"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Collection",type="write"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Database",type="read"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Database",type="write"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Global",type="read"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Global",type="write"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Metadata",type="read"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Metadata",type="write"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="oplog",type="read"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="oplog",type="write"} 0 # HELP mongodb_mongod_locks_time_locked_local_microseconds_total amount of time in microseconds that any database has held the local lock # TYPE mongodb_mongod_locks_time_locked_local_microseconds_total counter mongodb_mongod_locks_time_locked_local_microseconds_total{database="Collection",type="read"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Collection",type="write"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Database",type="read"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Database",type="write"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Global",type="read"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Global",type="write"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Metadata",type="read"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Metadata",type="write"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="oplog",type="read"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="oplog",type="write"} 0 # HELP mongodb_mongod_memory The mem data structure holds information regarding the target system architecture of mongod and current memory use # TYPE mongodb_mongod_memory gauge mongodb_mongod_memory{type="mapped"} 0 mongodb_mongod_memory{type="mapped_with_journal"} 0 mongodb_mongod_memory{type="resident"} 205 mongodb_mongod_memory{type="virtual"} 1064 # HELP mongodb_mongod_metrics_cursor_open The open is an embedded document that contains data regarding open cursors # TYPE mongodb_mongod_metrics_cursor_open gauge mongodb_mongod_metrics_cursor_open{state="noTimeout"} 0 mongodb_mongod_metrics_cursor_open{state="pinned"} 1 mongodb_mongod_metrics_cursor_open{state="total"} 1 # HELP mongodb_mongod_metrics_cursor_timed_out_total timedOut provides the total number of cursors that have timed out since the server process started. If this number is large or growing at a regular rate, this may indicate an application error # TYPE mongodb_mongod_metrics_cursor_timed_out_total counter mongodb_mongod_metrics_cursor_timed_out_total 0 # HELP mongodb_mongod_metrics_document_total The document holds a document of that reflect document access and modification patterns and data use. Compare these values to the data in the opcounters document, which track total number of operations # TYPE mongodb_mongod_metrics_document_total counter mongodb_mongod_metrics_document_total{state="deleted"} 0 mongodb_mongod_metrics_document_total{state="inserted"} 140396 mongodb_mongod_metrics_document_total{state="returned"} 74105 mongodb_mongod_metrics_document_total{state="updated"} 0 # HELP mongodb_mongod_metrics_get_last_error_wtime_num_total num reports the total number of getLastError operations with a specified write concern (i.e. w) that wait for one or more members of a replica set to acknowledge the write operation (i.e. a w value greater than 1.) # TYPE mongodb_mongod_metrics_get_last_error_wtime_num_total gauge mongodb_mongod_metrics_get_last_error_wtime_num_total 0 # HELP mongodb_mongod_metrics_get_last_error_wtime_total_milliseconds total_millis reports the total amount of time in milliseconds that the mongod has spent performing getLastError operations with write concern (i.e. w) that wait for one or more members of a replica set to acknowledge the write operation (i.e. a w value greater than 1.) # TYPE mongodb_mongod_metrics_get_last_error_wtime_total_milliseconds counter mongodb_mongod_metrics_get_last_error_wtime_total_milliseconds 0 # HELP mongodb_mongod_metrics_get_last_error_wtimeouts_total wtimeouts reports the number of times that write concern operations have timed out as a result of the wtimeout threshold to getLastError. # TYPE mongodb_mongod_metrics_get_last_error_wtimeouts_total counter mongodb_mongod_metrics_get_last_error_wtimeouts_total 0 # HELP mongodb_mongod_metrics_operation_total operation is a sub-document that holds counters for several types of update and query operations that MongoDB handles using special operation types # TYPE mongodb_mongod_metrics_operation_total counter mongodb_mongod_metrics_operation_total{type="fastmod"} 0 mongodb_mongod_metrics_operation_total{type="idhack"} 0 mongodb_mongod_metrics_operation_total{type="scan_and_order"} 0 # HELP mongodb_mongod_metrics_query_executor_total queryExecutor is a document that reports data from the query execution system # TYPE mongodb_mongod_metrics_query_executor_total counter mongodb_mongod_metrics_query_executor_total{state="scanned"} 0 mongodb_mongod_metrics_query_executor_total{state="scanned_objects"} 74105 # HELP mongodb_mongod_metrics_record_moves_total moves reports the total number of times documents move within the on-disk representation of the MongoDB data set. Documents move as a result of operations that increase the size of the document beyond their allocated record size # TYPE mongodb_mongod_metrics_record_moves_total counter mongodb_mongod_metrics_record_moves_total 0 # HELP mongodb_mongod_metrics_repl_apply_batches_num_total num reports the total number of batches applied across all databases # TYPE mongodb_mongod_metrics_repl_apply_batches_num_total counter mongodb_mongod_metrics_repl_apply_batches_num_total 0 # HELP mongodb_mongod_metrics_repl_apply_batches_total_milliseconds total_millis reports the total amount of time the mongod has spent applying operations from the oplog # TYPE mongodb_mongod_metrics_repl_apply_batches_total_milliseconds counter mongodb_mongod_metrics_repl_apply_batches_total_milliseconds 0 # HELP mongodb_mongod_metrics_repl_apply_ops_total ops reports the total number of oplog operations applied # TYPE mongodb_mongod_metrics_repl_apply_ops_total counter mongodb_mongod_metrics_repl_apply_ops_total 0 # HELP mongodb_mongod_metrics_repl_buffer_count count reports the current number of operations in the oplog buffer # TYPE mongodb_mongod_metrics_repl_buffer_count gauge mongodb_mongod_metrics_repl_buffer_count 0 # HELP mongodb_mongod_metrics_repl_buffer_max_size_bytes maxSizeBytes reports the maximum size of the buffer. This value is a constant setting in the mongod, and is not configurable # TYPE mongodb_mongod_metrics_repl_buffer_max_size_bytes counter mongodb_mongod_metrics_repl_buffer_max_size_bytes 2.68435456e+08 # HELP mongodb_mongod_metrics_repl_buffer_size_bytes sizeBytes reports the current size of the contents of the oplog buffer # TYPE mongodb_mongod_metrics_repl_buffer_size_bytes gauge mongodb_mongod_metrics_repl_buffer_size_bytes 0 # HELP mongodb_mongod_metrics_repl_network_bytes_total bytes reports the total amount of data read from the replication sync source # TYPE mongodb_mongod_metrics_repl_network_bytes_total counter mongodb_mongod_metrics_repl_network_bytes_total 0 # HELP mongodb_mongod_metrics_repl_network_getmores_num_total num reports the total number of getmore operations, which are operations that request an additional set of operations from the replication sync source. # TYPE mongodb_mongod_metrics_repl_network_getmores_num_total counter mongodb_mongod_metrics_repl_network_getmores_num_total 0 # HELP mongodb_mongod_metrics_repl_network_getmores_total_milliseconds total_millis reports the total amount of time required to collect data from getmore operations # TYPE mongodb_mongod_metrics_repl_network_getmores_total_milliseconds counter mongodb_mongod_metrics_repl_network_getmores_total_milliseconds 0 # HELP mongodb_mongod_metrics_repl_network_ops_total ops reports the total number of operations read from the replication source. # TYPE mongodb_mongod_metrics_repl_network_ops_total counter mongodb_mongod_metrics_repl_network_ops_total 0 # HELP mongodb_mongod_metrics_repl_network_readers_created_total readersCreated reports the total number of oplog query processes created. MongoDB will create a new oplog query any time an error occurs in the connection, including a timeout, or a network operation. Furthermore, readersCreated will increment every time MongoDB selects a new source fore replication. # TYPE mongodb_mongod_metrics_repl_network_readers_created_total counter mongodb_mongod_metrics_repl_network_readers_created_total 1 # HELP mongodb_mongod_metrics_repl_oplog_insert_bytes_total insertBytes the total size of documents inserted into the oplog. # TYPE mongodb_mongod_metrics_repl_oplog_insert_bytes_total counter mongodb_mongod_metrics_repl_oplog_insert_bytes_total 0 # HELP mongodb_mongod_metrics_repl_oplog_insert_num_total num reports the total number of items inserted into the oplog. # TYPE mongodb_mongod_metrics_repl_oplog_insert_num_total counter mongodb_mongod_metrics_repl_oplog_insert_num_total 0 # HELP mongodb_mongod_metrics_repl_oplog_insert_total_milliseconds total_millis reports the total amount of time spent for the mongod to insert data into the oplog. # TYPE mongodb_mongod_metrics_repl_oplog_insert_total_milliseconds counter mongodb_mongod_metrics_repl_oplog_insert_total_milliseconds 0 # HELP mongodb_mongod_metrics_repl_preload_docs_num_total num reports the total number of documents loaded during the pre-fetch stage of replication # TYPE mongodb_mongod_metrics_repl_preload_docs_num_total counter mongodb_mongod_metrics_repl_preload_docs_num_total 0 # HELP mongodb_mongod_metrics_repl_preload_docs_total_milliseconds total_millis reports the total amount of time spent loading documents as part of the pre-fetch stage of replication # TYPE mongodb_mongod_metrics_repl_preload_docs_total_milliseconds counter mongodb_mongod_metrics_repl_preload_docs_total_milliseconds 0 # HELP mongodb_mongod_metrics_repl_preload_indexes_num_total num reports the total number of index entries loaded by members before updating documents as part of the pre-fetch stage of replication # TYPE mongodb_mongod_metrics_repl_preload_indexes_num_total counter mongodb_mongod_metrics_repl_preload_indexes_num_total 0 # HELP mongodb_mongod_metrics_repl_preload_indexes_total_milliseconds total_millis reports the total amount of time spent loading index entries as part of the pre-fetch stage of replication # TYPE mongodb_mongod_metrics_repl_preload_indexes_total_milliseconds counter mongodb_mongod_metrics_repl_preload_indexes_total_milliseconds 0 # HELP mongodb_mongod_metrics_storage_freelist_search_total metrics about searching records in the database. # TYPE mongodb_mongod_metrics_storage_freelist_search_total counter mongodb_mongod_metrics_storage_freelist_search_total{type="bucket_exhausted"} 0 mongodb_mongod_metrics_storage_freelist_search_total{type="requests"} 0 mongodb_mongod_metrics_storage_freelist_search_total{type="scanned"} 0 # HELP mongodb_mongod_metrics_ttl_deleted_documents_total deletedDocuments reports the total number of documents deleted from collections with a ttl index. # TYPE mongodb_mongod_metrics_ttl_deleted_documents_total counter mongodb_mongod_metrics_ttl_deleted_documents_total 0 # HELP mongodb_mongod_metrics_ttl_passes_total passes reports the number of times the background process removes documents from collections with a ttl index # TYPE mongodb_mongod_metrics_ttl_passes_total counter mongodb_mongod_metrics_ttl_passes_total 0 # HELP mongodb_mongod_network_bytes_total The network data structure contains data regarding MongoDB’s network use # TYPE mongodb_mongod_network_bytes_total counter mongodb_mongod_network_bytes_total{state="in_bytes"} 1.323188842e+09 mongodb_mongod_network_bytes_total{state="out_bytes"} 1.401963024e+09 # HELP mongodb_mongod_network_metrics_num_requests_total The numRequests field is a counter of the total number of distinct requests that the server has received. Use this value to provide context for the bytesIn and bytesOut values to ensure that MongoDB’s network utilization is consistent with expectations and application use # TYPE mongodb_mongod_network_metrics_num_requests_total counter mongodb_mongod_network_metrics_num_requests_total 313023 # HELP mongodb_mongod_op_counters_repl_total The opcountersRepl data structure, similar to the opcounters data structure, provides an overview of database replication operations by type and makes it possible to analyze the load on the replica in more granular manner. These values only appear when the current host has replication enabled # TYPE mongodb_mongod_op_counters_repl_total counter mongodb_mongod_op_counters_repl_total{type="command"} 0 mongodb_mongod_op_counters_repl_total{type="delete"} 0 mongodb_mongod_op_counters_repl_total{type="getmore"} 0 mongodb_mongod_op_counters_repl_total{type="insert"} 0 mongodb_mongod_op_counters_repl_total{type="query"} 0 mongodb_mongod_op_counters_repl_total{type="update"} 0 # HELP mongodb_mongod_op_counters_total The opcounters data structure provides an overview of database operations by type and makes it possible to analyze the load on the database in more granular manner. These numbers will grow over time and in response to database use. Analyze these values over time to track database utilization # TYPE mongodb_mongod_op_counters_total counter mongodb_mongod_op_counters_total{type="command"} 164008 mongodb_mongod_op_counters_total{type="delete"} 0 mongodb_mongod_op_counters_total{type="getmore"} 74912 mongodb_mongod_op_counters_total{type="insert"} 70198 mongodb_mongod_op_counters_total{type="query"} 3907 mongodb_mongod_op_counters_total{type="update"} 0 # HELP mongodb_mongod_replset_heatbeat_interval_millis The frequency in milliseconds of the heartbeats # TYPE mongodb_mongod_replset_heatbeat_interval_millis gauge mongodb_mongod_replset_heatbeat_interval_millis{set="test1"} 2000 # HELP mongodb_mongod_replset_member_config_version The configVersion value is the replica set configuration version. # TYPE mongodb_mongod_replset_member_config_version gauge mongodb_mongod_replset_member_config_version{name="localhost:27017",set="test1",state="PRIMARY"} 2 mongodb_mongod_replset_member_config_version{name="localhost:27027",set="test1",state="SECONDARY"} 2 # HELP mongodb_mongod_replset_member_election_date The timestamp the node was elected as replica leader # TYPE mongodb_mongod_replset_member_election_date gauge mongodb_mongod_replset_member_election_date{name="localhost:27017",set="test1",state="PRIMARY"} 1.488818198e+09 # HELP mongodb_mongod_replset_member_health This field conveys if the member is up (1) or down (0). # TYPE mongodb_mongod_replset_member_health gauge mongodb_mongod_replset_member_health{name="localhost:27017",set="test1",state="PRIMARY"} 1 mongodb_mongod_replset_member_health{name="localhost:27027",set="test1",state="SECONDARY"} 1 # HELP mongodb_mongod_replset_member_last_heartbeat The lastHeartbeat value provides an ISODate formatted date and time of the transmission time of last heartbeat received from this member # TYPE mongodb_mongod_replset_member_last_heartbeat gauge mongodb_mongod_replset_member_last_heartbeat{name="localhost:27027",set="test1",state="SECONDARY"} 1.488834065e+09 # HELP mongodb_mongod_replset_member_last_heartbeat_recv The lastHeartbeatRecv value provides an ISODate formatted date and time that the last heartbeat was received from this member # TYPE mongodb_mongod_replset_member_last_heartbeat_recv gauge mongodb_mongod_replset_member_last_heartbeat_recv{name="localhost:27027",set="test1",state="SECONDARY"} 1.488834066e+09 # HELP mongodb_mongod_replset_member_optime_date The timestamp of the last oplog entry that this member applied. # TYPE mongodb_mongod_replset_member_optime_date gauge mongodb_mongod_replset_member_optime_date{name="localhost:27017",set="test1",state="PRIMARY"} 1.488833953e+09 mongodb_mongod_replset_member_optime_date{name="localhost:27027",set="test1",state="SECONDARY"} 1.488833953e+09 # HELP mongodb_mongod_replset_member_ping_ms The pingMs represents the number of milliseconds (ms) that a round-trip packet takes to travel between the remote member and the local instance. # TYPE mongodb_mongod_replset_member_ping_ms gauge mongodb_mongod_replset_member_ping_ms{name="localhost:27027",set="test1",state="SECONDARY"} 0 # HELP mongodb_mongod_replset_member_state The value of state is an integer between 0 and 10 that represents the replica state of the member. # TYPE mongodb_mongod_replset_member_state gauge mongodb_mongod_replset_member_state{name="localhost:27017",set="test1",state="PRIMARY"} 1 mongodb_mongod_replset_member_state{name="localhost:27027",set="test1",state="SECONDARY"} 2 # HELP mongodb_mongod_replset_member_uptime The uptime field holds a value that reflects the number of seconds that this member has been online. # TYPE mongodb_mongod_replset_member_uptime counter mongodb_mongod_replset_member_uptime{name="localhost:27017",set="test1",state="PRIMARY"} 15881 mongodb_mongod_replset_member_uptime{name="localhost:27027",set="test1",state="SECONDARY"} 15855 # HELP mongodb_mongod_replset_my_name The replica state name of the current member # TYPE mongodb_mongod_replset_my_name gauge mongodb_mongod_replset_my_name{name="localhost:27017",set="test1"} 1 # HELP mongodb_mongod_replset_my_state An integer between 0 and 10 that represents the replica state of the current member # TYPE mongodb_mongod_replset_my_state gauge mongodb_mongod_replset_my_state{set="test1"} 1 # HELP mongodb_mongod_replset_number_of_members The number of replica set mebers # TYPE mongodb_mongod_replset_number_of_members gauge mongodb_mongod_replset_number_of_members{set="test1"} 2 # HELP mongodb_mongod_replset_oplog_head_timestamp The timestamp of the newest change in the oplog # TYPE mongodb_mongod_replset_oplog_head_timestamp gauge mongodb_mongod_replset_oplog_head_timestamp 1.488833953e+09 # HELP mongodb_mongod_replset_oplog_items_total The total number of changes in the oplog # TYPE mongodb_mongod_replset_oplog_items_total gauge mongodb_mongod_replset_oplog_items_total 70202 # HELP mongodb_mongod_replset_oplog_size_bytes Size of oplog in bytes # TYPE mongodb_mongod_replset_oplog_size_bytes gauge mongodb_mongod_replset_oplog_size_bytes{type="current"} 1.273140034e+09 mongodb_mongod_replset_oplog_size_bytes{type="storage"} 1.273139968e+09 # HELP mongodb_mongod_replset_oplog_tail_timestamp The timestamp of the oldest change in the oplog # TYPE mongodb_mongod_replset_oplog_tail_timestamp gauge mongodb_mongod_replset_oplog_tail_timestamp 1.488818198e+09 # HELP mongodb_mongod_replset_term The election count for the replica set, as known to this replica set member # TYPE mongodb_mongod_replset_term gauge mongodb_mongod_replset_term{set="test1"} 1 # HELP mongodb_mongod_rocksdb_background_errors The total number of background errors in RocksDB # TYPE mongodb_mongod_rocksdb_background_errors gauge mongodb_mongod_rocksdb_background_errors 0 # HELP mongodb_mongod_rocksdb_block_cache_bytes The current bytes used in the RocksDB Block Cache # TYPE mongodb_mongod_rocksdb_block_cache_bytes gauge mongodb_mongod_rocksdb_block_cache_bytes 2.5165824e+07 # HELP mongodb_mongod_rocksdb_block_cache_hits_total The total number of hits to the RocksDB Block Cache # TYPE mongodb_mongod_rocksdb_block_cache_hits_total counter mongodb_mongod_rocksdb_block_cache_hits_total 519483 # HELP mongodb_mongod_rocksdb_block_cache_misses_total The total number of misses to the RocksDB Block Cache # TYPE mongodb_mongod_rocksdb_block_cache_misses_total counter mongodb_mongod_rocksdb_block_cache_misses_total 219214 # HELP mongodb_mongod_rocksdb_bloom_filter_useful_total The total number of times the RocksDB Bloom Filter was useful # TYPE mongodb_mongod_rocksdb_bloom_filter_useful_total counter mongodb_mongod_rocksdb_bloom_filter_useful_total 130251 # HELP mongodb_mongod_rocksdb_bytes_read_total The total number of bytes read by RocksDB # TYPE mongodb_mongod_rocksdb_bytes_read_total counter mongodb_mongod_rocksdb_bytes_read_total{type="compation"} 7.917332498e+09 mongodb_mongod_rocksdb_bytes_read_total{type="iteration"} 3.94838428e+09 mongodb_mongod_rocksdb_bytes_read_total{type="point_lookup"} 2.655612381e+09 # HELP mongodb_mongod_rocksdb_bytes_written_total The total number of bytes written by RocksDB # TYPE mongodb_mongod_rocksdb_bytes_written_total counter mongodb_mongod_rocksdb_bytes_written_total{type="compaction"} 9.531638929e+09 mongodb_mongod_rocksdb_bytes_written_total{type="flush"} 2.525874187e+09 mongodb_mongod_rocksdb_bytes_written_total{type="total"} 2.548843861e+09 # HELP mongodb_mongod_rocksdb_compaction_average_seconds The average time per compaction between levels N and N+1 in RocksDB # TYPE mongodb_mongod_rocksdb_compaction_average_seconds gauge mongodb_mongod_rocksdb_compaction_average_seconds{level="L0"} 0.505 mongodb_mongod_rocksdb_compaction_average_seconds{level="L5"} 3.606 mongodb_mongod_rocksdb_compaction_average_seconds{level="L6"} 14.579 mongodb_mongod_rocksdb_compaction_average_seconds{level="total"} 4.583 # HELP mongodb_mongod_rocksdb_compaction_bytes_per_second The rate at which data is processed during compaction between levels N and N+1 in RocksDB # TYPE mongodb_mongod_rocksdb_compaction_bytes_per_second gauge mongodb_mongod_rocksdb_compaction_bytes_per_second{level="L0",type="write"} 1.248854016e+08 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="L5",type="read"} 7.45537536e+07 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="L5",type="write"} 7.45537536e+07 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="L6",type="read"} 2.44318208e+07 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="L6",type="write"} 2.06569472e+07 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="total",type="read"} 2.70532608e+07 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="total",type="write"} 3.2505856e+07 # HELP mongodb_mongod_rocksdb_compaction_file_threads The number of threads currently doing compaction for levels in RocksDB # TYPE mongodb_mongod_rocksdb_compaction_file_threads gauge mongodb_mongod_rocksdb_compaction_file_threads{level="L0"} 0 mongodb_mongod_rocksdb_compaction_file_threads{level="L5"} 0 mongodb_mongod_rocksdb_compaction_file_threads{level="L6"} 0 mongodb_mongod_rocksdb_compaction_file_threads{level="total"} 0 # HELP mongodb_mongod_rocksdb_compaction_score The compaction score of RocksDB levels # TYPE mongodb_mongod_rocksdb_compaction_score gauge mongodb_mongod_rocksdb_compaction_score{level="L0"} 0 mongodb_mongod_rocksdb_compaction_score{level="L5"} 0.9 mongodb_mongod_rocksdb_compaction_score{level="L6"} 0 mongodb_mongod_rocksdb_compaction_score{level="total"} 0 # HELP mongodb_mongod_rocksdb_compaction_seconds_total The time spent doing compactions between levels N and N+1 in RocksDB # TYPE mongodb_mongod_rocksdb_compaction_seconds_total counter mongodb_mongod_rocksdb_compaction_seconds_total{level="L0"} 20 mongodb_mongod_rocksdb_compaction_seconds_total{level="L5"} 25 mongodb_mongod_rocksdb_compaction_seconds_total{level="L6"} 248 mongodb_mongod_rocksdb_compaction_seconds_total{level="total"} 293 # HELP mongodb_mongod_rocksdb_compaction_write_amplification The write amplification factor from compaction between levels N and N+1 in RocksDB # TYPE mongodb_mongod_rocksdb_compaction_write_amplification gauge mongodb_mongod_rocksdb_compaction_write_amplification{level="L5"} 1 mongodb_mongod_rocksdb_compaction_write_amplification{level="L6"} 1.7 mongodb_mongod_rocksdb_compaction_write_amplification{level="total"} 3.8 # HELP mongodb_mongod_rocksdb_compactions_total The total number of compactions between levels N and N+1 in RocksDB # TYPE mongodb_mongod_rocksdb_compactions_total counter mongodb_mongod_rocksdb_compactions_total{level="L0"} 40 mongodb_mongod_rocksdb_compactions_total{level="L5"} 7 mongodb_mongod_rocksdb_compactions_total{level="L6"} 17 mongodb_mongod_rocksdb_compactions_total{level="total"} 64 # HELP mongodb_mongod_rocksdb_estimate_table_readers_memory_bytes The estimate RocksDB table-reader memory bytes # TYPE mongodb_mongod_rocksdb_estimate_table_readers_memory_bytes gauge mongodb_mongod_rocksdb_estimate_table_readers_memory_bytes 2.10944e+06 # HELP mongodb_mongod_rocksdb_files The number of files in a RocksDB level # TYPE mongodb_mongod_rocksdb_files gauge mongodb_mongod_rocksdb_files{level="L0"} 0 mongodb_mongod_rocksdb_files{level="L5"} 2 mongodb_mongod_rocksdb_files{level="L6"} 23 mongodb_mongod_rocksdb_files{level="total"} 25 # HELP mongodb_mongod_rocksdb_immutable_memtables The total number of immutable MemTables in RocksDB # TYPE mongodb_mongod_rocksdb_immutable_memtables gauge mongodb_mongod_rocksdb_immutable_memtables 0 # HELP mongodb_mongod_rocksdb_iterations_total The total number of iterations performed by RocksDB # TYPE mongodb_mongod_rocksdb_iterations_total counter mongodb_mongod_rocksdb_iterations_total{type="backward"} 1955 mongodb_mongod_rocksdb_iterations_total{type="forward"} 206563 # HELP mongodb_mongod_rocksdb_keys_total The total number of RocksDB key operations # TYPE mongodb_mongod_rocksdb_keys_total counter mongodb_mongod_rocksdb_keys_total{type="read"} 209546 mongodb_mongod_rocksdb_keys_total{type="written"} 281332 # HELP mongodb_mongod_rocksdb_live_versions The current number of live versions in RocksDB # TYPE mongodb_mongod_rocksdb_live_versions gauge mongodb_mongod_rocksdb_live_versions 1 # HELP mongodb_mongod_rocksdb_memtable_bytes The current number of MemTable bytes in RocksDB # TYPE mongodb_mongod_rocksdb_memtable_bytes gauge mongodb_mongod_rocksdb_memtable_bytes{type="active"} 2.5165824e+07 mongodb_mongod_rocksdb_memtable_bytes{type="total"} 2.5165824e+07 # HELP mongodb_mongod_rocksdb_memtable_entries The current number of Memtable entries in RocksDB # TYPE mongodb_mongod_rocksdb_memtable_entries gauge mongodb_mongod_rocksdb_memtable_entries{type="active"} 4553 mongodb_mongod_rocksdb_memtable_entries{type="immutable"} 0 # HELP mongodb_mongod_rocksdb_oldest_snapshot_timestamp The timestamp of the oldest snapshot in RocksDB # TYPE mongodb_mongod_rocksdb_oldest_snapshot_timestamp gauge mongodb_mongod_rocksdb_oldest_snapshot_timestamp 0 # HELP mongodb_mongod_rocksdb_pending_compactions The total number of compactions pending in RocksDB # TYPE mongodb_mongod_rocksdb_pending_compactions gauge mongodb_mongod_rocksdb_pending_compactions 0 # HELP mongodb_mongod_rocksdb_pending_memtable_flushes The total number of MemTable flushes pending in RocksDB # TYPE mongodb_mongod_rocksdb_pending_memtable_flushes gauge mongodb_mongod_rocksdb_pending_memtable_flushes 0 # HELP mongodb_mongod_rocksdb_read_latency_microseconds The read latency in RocksDB in microseconds by level # TYPE mongodb_mongod_rocksdb_read_latency_microseconds gauge mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="P50"} 9.33 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="P75"} 17.03 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="P99"} 1972.31 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="P99.9"} 9061.44 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="P99.99"} 18098.3 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="avg"} 87.3861 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="max"} 26434 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="median"} 9.3328 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="min"} 0 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="stddev"} 607.16 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="P50"} 10.85 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="P75"} 28.46 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="P99"} 4208.33 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="P99.9"} 13079.55 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="P99.99"} 23156.25 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="avg"} 216.3736 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="max"} 34585 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="median"} 10.8461 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="min"} 0 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="stddev"} 978.48 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="P50"} 9.42 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="P75"} 30.69 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="P99"} 6037.87 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="P99.9"} 15482.63 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="P99.99"} 30125.5 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="avg"} 294.9574 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="max"} 61051 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="median"} 9.4245 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="min"} 0 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="stddev"} 1264.12 # HELP mongodb_mongod_rocksdb_reads_total The total number of read operations in RocksDB # TYPE mongodb_mongod_rocksdb_reads_total counter mongodb_mongod_rocksdb_reads_total{level="L0"} 59017 mongodb_mongod_rocksdb_reads_total{level="L5"} 42125 mongodb_mongod_rocksdb_reads_total{level="L6"} 118745 # HELP mongodb_mongod_rocksdb_seeks_total The total number of seeks performed by RocksDB # TYPE mongodb_mongod_rocksdb_seeks_total counter mongodb_mongod_rocksdb_seeks_total 156184 # HELP mongodb_mongod_rocksdb_size_bytes The total byte size of levels in RocksDB # TYPE mongodb_mongod_rocksdb_size_bytes gauge mongodb_mongod_rocksdb_size_bytes{level="L0"} 0 mongodb_mongod_rocksdb_size_bytes{level="L5"} 1.2969836544e+08 mongodb_mongod_rocksdb_size_bytes{level="L6"} 1.46175688704e+09 mongodb_mongod_rocksdb_size_bytes{level="total"} 1.59145525248e+09 # HELP mongodb_mongod_rocksdb_snapshots The current number of snapshots in RocksDB # TYPE mongodb_mongod_rocksdb_snapshots gauge mongodb_mongod_rocksdb_snapshots 0 # HELP mongodb_mongod_rocksdb_stall_percent The percentage of time RocksDB has been stalled # TYPE mongodb_mongod_rocksdb_stall_percent gauge mongodb_mongod_rocksdb_stall_percent 0 # HELP mongodb_mongod_rocksdb_stalled_seconds_total The total number of seconds RocksDB has spent stalled # TYPE mongodb_mongod_rocksdb_stalled_seconds_total counter mongodb_mongod_rocksdb_stalled_seconds_total 0 # HELP mongodb_mongod_rocksdb_stalls_total The total number of stalls in RocksDB # TYPE mongodb_mongod_rocksdb_stalls_total counter mongodb_mongod_rocksdb_stalls_total{type="level0_numfiles"} 0 mongodb_mongod_rocksdb_stalls_total{type="level0_numfiles_with_compaction"} 0 mongodb_mongod_rocksdb_stalls_total{type="level0_slowdown"} 0 mongodb_mongod_rocksdb_stalls_total{type="level0_slowdown_with_compaction"} 0 mongodb_mongod_rocksdb_stalls_total{type="memtable_compaction"} 0 mongodb_mongod_rocksdb_stalls_total{type="memtable_slowdown"} 0 # HELP mongodb_mongod_rocksdb_total_live_recovery_units The total number of live recovery units in RocksDB # TYPE mongodb_mongod_rocksdb_total_live_recovery_units gauge mongodb_mongod_rocksdb_total_live_recovery_units 5 # HELP mongodb_mongod_rocksdb_transaction_engine_keys The current number of transaction engine keys in RocksDB # TYPE mongodb_mongod_rocksdb_transaction_engine_keys gauge mongodb_mongod_rocksdb_transaction_engine_keys 0 # HELP mongodb_mongod_rocksdb_transaction_engine_snapshots The current number of transaction engine snapshots in RocksDB # TYPE mongodb_mongod_rocksdb_transaction_engine_snapshots gauge mongodb_mongod_rocksdb_transaction_engine_snapshots 0 # HELP mongodb_mongod_rocksdb_write_ahead_log_bytes_per_second The number of bytes written per second by the Write-Ahead-Log in RocksDB # TYPE mongodb_mongod_rocksdb_write_ahead_log_bytes_per_second gauge mongodb_mongod_rocksdb_write_ahead_log_bytes_per_second 157286.4 # HELP mongodb_mongod_rocksdb_write_ahead_log_writes_per_sync The number of writes per Write-Ahead-Log sync in RocksDB # TYPE mongodb_mongod_rocksdb_write_ahead_log_writes_per_sync gauge mongodb_mongod_rocksdb_write_ahead_log_writes_per_sync 70538 # HELP mongodb_mongod_rocksdb_writes_per_batch The number of writes per batch in RocksDB # TYPE mongodb_mongod_rocksdb_writes_per_batch gauge mongodb_mongod_rocksdb_writes_per_batch 2.54476812288e+09 # HELP mongodb_mongod_rocksdb_writes_per_second The number of writes per second in RocksDB # TYPE mongodb_mongod_rocksdb_writes_per_second gauge mongodb_mongod_rocksdb_writes_per_second 157286.4 # HELP mongodb_mongod_storage_engine The storage engine used by the MongoDB instance # TYPE mongodb_mongod_storage_engine counter mongodb_mongod_storage_engine{engine="rocksdb"} 1

Grafana

PMM uses Grafana to visualize metrics stored in Prometheus. Grafana uses a concept of “dashboards” to store the definitions of what it should visualize. PMM’s dashboards are hosted under the GitHub project: percona/grafana-dashboards.

Grafana can support multiple “data sources” for metrics. As Percona Monitoring and Management uses Prometheus for metric storage, this is the “data source” used in Grafana.

Adding Custom Graphs to Grafana

In this section I will create an example graph to help indicate the “efficiency” of queries on a Mongod instance over the last 5 minutes. This is a good example to use because we plan to add this exact metric in an upcoming version of PMM.

This graph will rely on two metrics that are already provided to PMM via percona/mongodb_exporter since at least version 1.0.0 (“$host” = the hostname of a given node – explained later):

  1. mongodb_mongod_metrics_query_executor_total{instance=”$host”, state=”scanned_objects”} – A metric representing the total number of documents scanned by the server.
  2. mongodb_mongod_metrics_document_total{instance=”$host”, state=”returned”} – A metric representing the total number of documents returned by the server.

The graph will compute the change in these two metrics over five minutes and create a percentage/ratio of scanned vs. returned documents. A host with ten scanned documents and one document returned would have 10% efficiency and a host that scanned 100 documents to return 100 documents would have 100% efficiency (a 1:1 ratio).

Often you will encounter Prometheus metrics that are “total” metric counters that increment from the time the server is (re)started. Both of the metrics our new graph requires are incremented counters and thus need to be “scaled” or “trimmed” to only show the last five minutes of metrics (in this example), not the total since the server was (re)started.

Prometheus offers a very useful query function for incremented counters called “increase()”: https://prometheus.io/docs/querying/functions/#increase(). The Prometheus increase() function allows queries to return the amount a metric counter has increased over a given time period, making this trivial to do! It is also unaffected by counter “resets” due to server restarts as increase() only returns increases in counters.

The increase() syntax requires a time range to be specified before the closing round-bracket. In our case we will as increase() to consider the last five minutes, which is expressed with “[5m]” at the end of the increase() function, seen in the following example.

The full Prometheus query I will use to create query efficiency graph is:

sum( increase(mongodb_mongod_metrics_query_executor_total{instance="$host", state="scanned_objects"}[5m]) ) / sum( increase(mongodb_mongod_metrics_document_total{instance="$host", state="returned"}[5m]) )

Note: sum() is required around the increase() functions when dividing two numbers in Prometheus queries.

Now, let’s make this a new graph! To do this you can create a new dashboard in PMM’s Grafana or add to an existing dashboard. In this example I’ll create a new dashboard with a single graph.

To add a new dashboard, press the Dashboard selector in PMM’s Grafana and select “Create New”:

This will create a new dashboard named “New Dashboard”.

Most of PMM’s graphing uses a variable named “$host” in place of a hostname/IP. You’ll notice “$host” was used in the “query efficiency” Prometheus query earlier. The variable is set using a Grafana feature called Templating.

Let’s add a “$host” variable to our new dashboard so we can change what host we graph without modifying our queries. First, press the gear icon at the top of the dashboard and select “Templating”:

Then press “New” to create a new Templating variable.

Set “Name” to be host, set “Data Source” to Prometheus and set “Query” to label_values(instance). Leave all other settings default:

Press “Add” to add the template variable, then save and reload the dashboard.

This will add a drop-down of unique hosts in Prometheus like this:

On the first dashboard row let’s add a new graph by opening the row menu on the far left of the first row and then select “Add Panel”:

Select “Graph” as the type. Click the title of the blank graph to open a menu, press “Edit”:

This opens Grafana’s graph editor with an empty graph and some input boxes seen below.

Next, let’s add our “query efficiency” Prometheus query (earlier in this article) to the “Query” input field and add a legend name to “Legend format”:

Now we have some graph data, but the Y-axis and title don’t explain very much about the metric. What does “1.0K” on the Y-Axis mean?

As our metric is a ratio, let’s display the Y-axis as a percentage by switching to the “Axes” tab, then selecting “percent (0.0-1.0)” as the “Unit” selection for the “Left Y” axis, like so:

Next let’s set a graph title. To do this, go to the “General” tab of the graph editor and set the “Title” field:

And now we have a “Query Efficiency” graph with an accurate Y-axis and Title(!):

“Back to Dashboard” on the top-right will take you back to the dashboard view. Always remember to save your dashboards after making changes!

Adding Custom Metrics to percona/mongodb_exporter

For those familiar with the Go programming language, adding custom metrics to percona/mongodb_exporter is fairly straightforward. The percona/mongodb_exporter uses the Prometheus Go client to export metrics gathered from queries to MongoDB.

Adding a completely new metric to the exporter is unfortunately too open-ended to explain in a blog. Instead, I will cover how an existing metric is exported by percona/mongodb_exporter. The process for a new metric will be similar.

To follow our previous example, here is an simplified example of how the metric: “mongodb_mongod_metrics_query_executor_total” is exported via percona/mongodb_exporter. This source of this metric is “db.serverStatus().metrics.queryExecutor” from MongoDB shell perspective.

First a new Prometheus metric is defined as a go ‘var’:

  1. var ( metricsQueryExecutorTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ Namespace: Namespace, Name: "metrics_query_executor_total", Help: "queryExecutor is a document that reports data from the query execution system", }, []string{"state"}) )
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/metrics.go#L58-L64
  2. A struct for marshaling the BSON response from MongoDB and an “.Export()” function is defined for the struct (.Export() is called on the metric structs):
    // QueryExecutorStats are the stats associated with a query execution. type QueryExecutorStats struct { Scanned float64 `bson:"scanned"` ScannedObjects float64 `bson:"scannedObjects"` } // Export exports the query executor stats. func (queryExecutorStats *QueryExecutorStats) Export(ch chan<- prometheus.Metric) { metricsQueryExecutorTotal.WithLabelValues("scanned").Set(queryExecutorStats.Scanned) metricsQueryExecutorTotal.WithLabelValues("scanned_objects").Set(queryExecutorStats.ScannedObjects) }
    Notice that the float64 values unmarshaled from the BSON are used in the .Set() for the Prometheus metric. All Prometheus values must be float64.
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/metrics.go#L271-L281
  3. In this case the “QueryExecutorStats” is a sub-struct of a larger “MetricStats” struct above it, also with its own “.Export()” function:
    // MetricsStats are all stats associated with metrics of the system type MetricsStats struct { Document *DocumentStats `bson:"document"` GetLastError *GetLastErrorStats `bson:"getLastError"` Operation *OperationStats `bson:"operation"` QueryExecutor *QueryExecutorStats `bson:"queryExecutor"` Record *RecordStats `bson:"record"` Repl *ReplStats `bson:"repl"` Storage *StorageStats `bson:"storage"` Cursor *CursorStats `bson:"cursor"` } // Export exports the metrics stats. func (metricsStats *MetricsStats) Export(ch chan<- prometheus.Metric) { if metricsStats.Document != nil { metricsStats.Document.Export(ch) } if metricsStats.GetLastError != nil { metricsStats.GetLastError.Export(ch) } if metricsStats.Operation != nil { metricsStats.Operation.Export(ch) } if metricsStats.QueryExecutor != nil { metricsStats.QueryExecutor.Export(ch) } ...
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/metrics.go#L405-L430
  4. Finally a .Collect() and  .Describe()” (also required functions) is called on the metric to collect and describe it:
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/metrics.go#L451
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/metrics.go#L485
  5. Later on in this code, “MetricStats” is passed the result of the query “db.serverStatus().metrics”. This can be seen at:
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/server_status.go#L60
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/server_status.go#L177-L179
    and
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/server_status.go#L197-L207

For those unfamiliar with Go and/or unable to contribute new metrics to the project, we suggest you open a JIRA ticket for any feature requests for new metrics here: https://jira.percona.com/projects/PMM.

Conclusion

With the flexibility of the monitoring components of Percona Monitoring and Management, the sky is the limit on what can be done with database monitoring! Hopefully this blog gives you a taste of what is possible if you need to add a new graph, a new metric or both to Percona Monitoring and Management. Also, it is worth repeating that a large number of metrics gathered in Percona Monitoring and Management are not graphed. Perhaps what you’re looking for is already collected. See “http://<pmm-server>/prometheus” for more details on what metrics are stored in Prometheus.

We are always open to improving our dashboards, and we would love to hear about any custom graphs you create and how they help solve problems!

Categories: MySQL

Percona Server for MongoDB 3.2.12-3.2 is now available

MySQL Performance Blog - Thu, 2017-03-09 18:52

Percona announces the release of Percona Server for MongoDB 3.2.12-3.2 on March 9, 2017. Download the latest version from the Percona web site or the Percona Software Repositories.

Percona Server for MongoDB 3.2.11-3.1 is an enhanced, open-source, fully compatible, highly scalable, zero-maintenance downtime database supporting the MongoDB v3.2 protocol and drivers. It extends MongoDB with MongoRocks, Percona Memory Engine, and PerconaFT storage engine, as well as enterprise-grade features like External Authentication, Audit Logging, Profiling Rate Limiting, and Hot Backup at no extra cost. Percona Server for MongoDB requires no changes to MongoDB applications or code.

NOTE: We deprecated the PerconaFT storage engine. It will not be available in future releases.

This release is based on MongoDB 3.2.12 and includes the following additional changes:

  • PSMDB-17: Changed welcome message in the shell to mention Percona Server for MongoDB instead of MongoDB
  • PSMDB-90: Added error message for storage engines that do not support Hot Backup
  • PSMDB-91: Deprecated audit configuration section and added auditLog instead
  • PSMDB-95: Fixed version dependencies for sub packages so that all corresponding packages get updated accordingly
  • PSMDB-96: Excluded diagnostic.data directory when using TokuBackup with PerconaFT
  • PSMDB-98: Improved Hot Backup to create destination folder if it does not exist
  • PSMDB-101: Implemented the auditAuthorizationSuccess parameter to enable auditing of authorization success
  • PSMDB-104: Updated links in client shell output to point to Percona’s documentation and forum
  • PSMDB-107: Fixed behavior when creating the audit log file
  • PSMDB-111: Refactored external_auth tests
  • PSMDB-123: Fixed the creation of proper subdirectories inside the backup destination directory
  • PSMDB-126: Added index and collection name to duplicate key error message
  • Fixed startup scripts for Ubuntu 14.04.5 LTS (Trusty Tahr)
  • Fixed a number of other small issues and bugs

Percona Server for MongoDB 3.2.12-3.2 release notes are available in the official documentation.

Categories: MySQL

Services Monitoring with Probabilistic Fault Detection

MySQL Performance Blog - Thu, 2017-03-09 18:19

In this blog post, we’ll discuss services monitoring using probabilistic fault detection.

Let’s admit it, the task of monitoring services is one of the most difficult. It is time-consuming, error-prone and difficult to automate. The usual monitoring approach has been pretty straightforward in the last few years: setup a service like Nagios, or pay money to get a cloud-based monitoring tool. Then choose the metrics you are interested in and set the thresholds. This is a manual process that works when you have a small number of services and servers, and you know exactly how they behave and what you should monitor. These days, we have hundred of servers with thousands of services sending us millions of metrics. That is the first problem: the manual approach to configuration doesn’t work.

That is not the only problem. We know that no two servers perform the same because no two servers have exactly the same workload. The thresholds that you setup for one server might not be the correct one for all of the other thousand. There are some approaches to the problem that will make it even worse (like taking averages and setting the thresholds based on those, for example, hoping it will work). Let me tell you a secret: it won’t work. Here we have a second problem: instances of the same type can demonstrate very different behaviors.

The last problem is that new shiny services you company may want to use are announced every week. It is impossible, because of time constraints, to know all of those services well enough to create a perfect monitoring template. In other words: sometimes we are asked to monitor software we don’t completely understand.

In summary, you have thousands of services, some of them you don’t even know how they work, that are sending you million of metrics that mean nothing to you. Now, set the thresholds and enable the pager alert. The nightmare has started. Is there a different approach?

Machine Learning

We have to stop thinking that monitoring is a bunch of config files with thresholds that we copy from one server to another. There are no magic templates that will work. We need to use a different technique that removes us from the process. That template is “machine learning.” As stated in Wikipedia, it is a subfield of computer science that gives computers the ability to learn without being explicitly programmed. In it’s most basic form, it can be used to solve classification problems. For example, open pet photos and identify if it is a cat or a dog. This is a classification problem that both humans and computers can solve, but we are much much slower. The computer has to take the time to learn the patterns, but at some point it will do the classification in no time.

I hope you are starting to see a pattern here. Why do we need to care about monitoring and its configuration if we have computers that can learn patterns and classify things for us?

There are two main ways of doing probabilistic fault detection: Novelty Detection and Outlier Detection.

Novelty Detection

Novelty Detection is easy to visualize and understand. It takes a series of inputs and tries to find anomalies, something that hasn’t been seen before. For example, our credit card company has a function that takes “category, expense, date, hour, country” as arguments and returns an integer so that they can classify and identify all the purchases. Your monthly use of the credit card looks like this:

[0,4,4,5,5,5,4,3]

That is the normal model that defines your use of the credit card. Now, it can be used to detect anomalies.

  • [0] – OK
  • [4] – OK
  • [4] – OK
  • [1] – Anomaly! Operation canceled.

Easy and straightforward. It is simple and very useful in a lot of areas to generate alerts when something anomalous happens. One of the machine learning models that can do this is One-Class Support Vector Machines, but since this is not the kind of fault detection we are looking for I won’t go into details. If you are interested, follow these links:

Outlier Detection

Let’s say we have this data:

[0, 3, 5, 1, -2, 19, 2, 10, -9, 2, 1, 8, 3, 21, -1, 3]

Now we know how to find anomalies, but how do we find outliers? Looking at the numbers above, it seems 21, 19 and -9 could be outliers. But a more exact definition is needed (not just intuition). The most simple and usual way of doing it is the following:

We divide our data into three pieces. One cut will be done at 25%, the second cut at 75%. The number that it is at 25% is called the First Quartile, and the value of the second cut is called the Third Quartile. The IQR or Interquartile Range is the subtraction of the Third Quartile with the First Quartile. Now, an outlier is any number that falls in one of these two categories:

  • If the value is below: (First Quartile) – (1.5 × IQR)
  • If the value is above: (Third Quartile + (1.5 × IQR)

Using Python:

inputs = [0, 3, 5, 1, -2, 19, 2, 10, -9, 2, 1, 8, 3, 21, -1, 3] Q1 = np.percentile(inputs,25) Q3 = np.percentile(inputs,75) step = (Q3-Q1)*1.5 Q1 0.75 Q3 5.75 step 7.5 outliers = [x for x in inputs if x < Q1-step or x > Q3+step] outliers [19, -9, 21]

This looks more like what we need. If we are monitoring a metric, and outliers are detected, then something is happening there that requires investigation. Some of the most used outlier detection models in scikit-learn are:

  • Elliptic Envelope: a robust co-variance estimate that assumes that our data is Gaussian distributed. It will define the shape of the data we have, creating a frontier that delimits the contour. As you probably guessed, it will be elliptical in shape. Don’t worry about the assumption of Gaussian distribution, data can be standardized. More about this later on.

 

  • Isolation Forest: this is the well-known “forest of random trees,” but applied to outlier detection. This is more suitable when we have many different input metrics. In the example I use below, I just use a single metric, so this model would not work that well.

Therefore, Elliptic Envelope looks like the best option for our proof-of-concept.

For visual reference, this is how the three models look like when they try to shape two data inputs:

Source: scikit-learn.org

 

Proof-of-Concept

I haven’t explained the model in detail, but a high level explanation should be enough to understand the problem and the possible solution. Let’s start building a proof-of-concept.

For this test, I got data from our Prometheus setup, where all the time-series monitoring data from our customers is stored. In this particular example, I got numbers from the “Threads Running” metric. Those will be used to train our Elliptical Envelope. It is important to take the following into account:

    • We need to collect enough data so that it captures the correct shape of our baseline performance. For example, usually nighttime hours have less of a workload than during the day (same with weekend days, in some cases).
    • As explained before, it assumes a Gaussian distribution, which means that the data needs to be scaled. I am going to standardize the data so that it has 0 mean and 1 variance. The same standardization needs to be applied to the data we test after the training process, when the monitoring is already in place. That standardization also needs to be applied to each metric individually. This is the formula:
Source: dataminingblog.com

With μ as the mean and σ as the standard deviation.

This is the summary of what our proof-of-concept will do:

  • Read Prometheus JSON dump.
  • Separate some data for training, standardizing it first.
  • Separate some data for testing, standardizing it first as well.
  • Make predictions on test data.
  • For those rows identified as outliers, get the original non-standardize data to see the number of threads running.

So, let’s start:

Import the Libraries

import pandas as pd import numpy as np import json from datetime import datetime from sklearn.preprocessing import StandardScaler from sklearn.covariance import EllipticEnvelope

Load the Data

All the information is in a JSON output from Prometheus that has the “threads_running” of a particular server. It has one second granularity for the last four weeks. I also converted “timestamps” to a normal “datetime” object so that it is easier to read:

with open('query.json') as data_file: data = json.load(data_file) data = pd.DataFrame(data["data"]["result"][0]["values"]) data[0] = data[0].astype(int) data[0] = data[0].apply(lambda x: datetime.fromtimestamp(x))

The data looks like this:

DateTime Threads Running 2017-01-19 20:32:44 1 2017-01-19 20:32:45 1 2017-01-19 20:32:46 2 … …

 

Create the Training and Testing Dataset

First, separate some of the data for use as training:

train_data = data[(data[0] >= "2017-01-22") & (data[0] <= "2017-01-28" )]

Ignore the date column, and just store the metrics:

train_data = train_data.iloc[:,[1]]

Standardize it:

escaler = StandardScaler() train_data = escaler.fit_transform(train_data)

Now the data looks like this:

Standardized Threads Running -0.4072634 -0.4072634 0.47153585 …

To create the test dataset we need to follow the exact same procedure, only select a different timeframe:

test_original_data = data[(data[0] >= "2017-02-2") & (data[0] <= "2017-02-17" )] test_data = test_original_data.iloc[:,[1]] test_data = escaler.transform(test_data)

Train the Model

Let’s create our model with the training data! I am using two parameters here:

  • assume_centered: to specify that our data is already Gaussian distributed.
  • contamination: to specify the ratio of outliers our training data has.

clf = EllipticEnvelope(assume_centered=True,contamination=0) clf.fit(train_data)

Search for Outliers

Now that we’ve trained the model and we have our test data, we can ask the model if it finds any outliers. It will return 1 or -1 for each row. “1” means that the value of threads running is normal and within the boundaries, while “-1” means that the value is an outlier:

predictions = clf.predict(test_data) outliers = np.where(predictions==-1)

The array “outliers” stores the row numbers where -1 was predicted.

At this point we have three important variables:

  • test_data: standardized testing data.
  • test_original_data: the original test data without modification.
  • outliers: the row numbers where an outlier was detected (-1).
Investigate the Outliers

Since we have the row number where an outlier was detected, now we can just query test_original_data and search for those rows. In this example, I show some random ones:

for indice in outliers[0]: if np.random.randn() > 2.5: print("{} - {} threads running".format(test_original_data.iloc[indice][0], test_original_data.iloc[indice][1])) 2017-02-03 11:26:03 - 41 threads running 2017-02-03 11:26:40 - 43 threads running 2017-02-03 11:27:50 - 48 threads running 2017-02-03 11:32:07 - 78 threads running 2017-02-03 11:33:25 - 90 threads running 2017-02-12 10:06:58 - 36 threads running 2017-02-12 10:12:11 - 60 threads running 2017-02-12 10:12:30 - 64 threads running

And there we have it! Dates and hours when something really out of the ordinary happened. No need to create a config file for each service, guess thresholds, adjust them … nothing. Just let the model learn, and you get alerts when something unexpected happens. Push all the metrics from your services to these models, and let them do the hard work.

Summary

Most companies have similar situations. Companies add new services on hundred of servers, and monitoring is an essential part of the infrastructure. The old method of monolithic config files with some thresholds doesn’t scale, because it needs a lot of manual work with a trial/error approach. The types of techniques explained in this blog post can help us deploy monitoring on hundred of servers, not really caring about the different nuances of each service or workload. It is even possible to start monitoring a service without even knowing anything about it — just let the probabilistic model take care of it.

It is important to clarify that, in my opinion, these fault detection models are not going to be a substitute for software like Nagios. In those areas where a binary test is needed (service is up/down for example), Nagios and other similar services do a good job. Actually, a Nagios check can use the procedure explained here. When there are many metrics to analyze, probabilistic methods can save us from a nightmare.

Categories: MySQL

Migrating MongoDB Away from MMAPv1

MySQL Performance Blog - Wed, 2017-03-08 23:16

This is another post in the series of blogs on the Percona Server for MongoDB 3.4 bundle release. In this blog post, we’ll discuss moving away from the MMAPv1 storage engine.

Introduction

WIth the MongoDB v3.0 release in February of 2015, the long-awaited ability to choose storage engines became a reality. As of version 3.0, you could choose two engines in MongoDB Community Server and, if you use Percona Server for MongoDB, you could choose from four. Here’s a table for ease of consumption:

Here’s a table for easy consumption:

Storage Engine Percona Server for MongoDB MongoDB Community Server MongoDB Enterprise Server (licensed) MMAPv1

WiredTiger

MongoRocks

In-memory

Encrypted

 

Why change engines?

With increased possibilities comes an increase in the decision-making process difficult (a concept that gets reinforced every time I take my mother out a restaurant with a large menu – ordering is never quick). In all seriousness, the introduction of the storage engine API to MongoDB is possibly the single greatest feature MongoDB, Inc has released to-date.

One of the biggest gripes from the pre-v3.0 days was MongoDB’s lack of scale. This was mostly due to the MMAPv1 storage engine, which suffered from a very primitive locking scheme. If you would like a illustration of the problem, think of the world’s biggest supermarket with one checkout line – you might be able to fit in lots of shoppers, but they’re not going to accomplish their goal quickly. So, the ability to increase performance and concurrency with a simple switch is huge! Additionally, modern storage engines support compression. This should reduce your space utilization when switching by at least 50%.

All the way up to MongoDB v3.2, the default storage engine was MMAPv1. If you didn’t make a conscious decision about what storage engine to choose when you started using MongoDB, there is a good chance that MMAPv1 is what you’re on. If you’d like to find out for sure what engine you’re using, simply run the command below. The output will be the name of the storage engine. As you can see, I was running the MMAPv1 storage engine on this machine. Now that we understand where we’re at, let’s get into where we can be in the future.

db.serverStatus().storageEngine.name mmapv1

Public Service Announcement

Before we get into what storage engine(s) to evaluate, we need to talk about testing. In my experience, a majority of the MySQL and MongoDB community members are rolling out changes to production without planning or testing. If you’re in the same boat, you’re in very good company (or at least in a great deal of company). However, you should stop this practice. It’s basic “sample size” in statistics – when engaged in risk-laden behavior, the optimal time to stop increasing the sample size is prior to the probability of failure reaching “1”. In other words, start your testing and planning process today!

At Percona, we recommend that you thoroughly test any database changes in a testing or development environment before you decide to roll them into production. Additionally, prior to rolling the changes into production (with a well thought out plan, of course), you’ll need to have a roll-back plan in case of unintended consequences. Luckily, with MongoDB’s built-in replication and election protocols, both are fairly easy. The key here is to plan. This is doubly true if you are undertaking a major version upgrade, or are jumping over major versions. With major version upgrades (or version jumps) comes the increased likelihood of a change in database behavior as it relates to your application’s response time (or even stability).

What should I think about?

In the table above, we listed the pre-packaged storage engine options that are available to us and other distributions. We also took a look at why you should consider moving off of MMAPv1 in the preceding section. To be clear, in my opinion a vast majority of MongoDB users that are on MMAPv1 can benefit from a switch. Which engine to switch to is the pressing question. Your first decision should be to evaluate whether or not your workload fits into the sweet spot for MMAPv1 by reading the section below. If that section doesn’t describe your application, then the additional sections should help you narrow down your choices.

Now, let’s take a look at what workloads match up with what storage engines.

MMAPv1

Believe it or not, there are some use cases where MMAPv1 is likely to give you as good (or better) performance as any other engine. If you’re not worried about the size of your database on disk, then you may not want to bother changing engines. Users that are likely to see no benefit from changing have read-heavy (or 100%) read applications. Also, certain update-heavy use cases, where you’re updating small amounts of data or performing $set operations, are likely to be faster on MMAPv1.

WiredTiger

WiredTiger is a the new default storage engine for MongoDB. It is a good option for general workloads that are currently running on MMAPv1. WiredTiger will give you good performance for most workloads and will reduce your storage utilization with compression that’s enabled by default. If you have a write-heavy workload, or are approaching high I/O utilization (>55%) with plans for it to rise, then you might benefit from a migration to WiredTiger.

MongoRocks (RocksDB from Facebook)

This is Facebook’s baby, which was forged in the fires of the former Parse business unit. MongoRocks, which uses LSM indexing, is advertised as “designed to work with fast storage.” Don’t let this claim fool you. For workloads that are heavy on writes, highly concurrent or approaching disk bound, MongoRocks could give you great benefits. In terms of compression, MongoRocks has the ability to efficiently handle deeper compression algorithms, which should further decrease your storage requirements.

In-Memory

The in-memory engine, whether we’re speaking about the MongoDB or Percona implementation, should be used for workloads where extreme low latency is the most important requirement. The types of applications that I’m talking about are usually low-latency, “real-time” apps — like decision making or user session tracking. The in-memory engine is not persistent, so it operates strictly out of the cache allocated to MongoDB. Consequently, the data may (and likely will) be lost if the server crashes.

Encrypted

This is for applications in highly secure environments where on-disk encryption is necessary for compliance. This engine will protect the MongoDB data in the case that a disk or server is stolen. On the flip side, this engine will not protect you from a hacker that has access to the server (MongoDB shell), or can intercept your application traffic. Another way to achieve the same level of encryption for compliance is using volume level encryption like LUKS. An additional benefit of volume level encryption, since it works outside the database, is re-use on all compliant servers (not just MongoDB).

Getting to your new engine

Switching to the new engine is actually pretty easy, especially if you’re running a replica set. One important caveat is that unlike MySQL, the storage engine can only be defined per mongod process (not per database or collection). This means that it’s an all or nothing operation on a single MongoDB process. You’ll need to reload your data on that server. This is necessary because the data files from one engine are not compatible with another engine. Thus reloading the data to transform from one engine format to another is necessary. Here are the high-level steps (assuming you’re running a replica set):

  1. Make sure you’re not in your production environment
  2. Backup your data (it can’t hurt)
  3. Remove a replica set member
  4. Rename (or delete) the old data directory. The member will re-sync with the replica set
    • Make sure you have enough disk space if you’re going to keep a copy of the old data directory
  5. Update the mongo.conf file to use a new storage engine. Here’s an example for RocksDB from our documentation:
    storage:  engine: rocksdb  rocksdb:    cacheSizeGB: 4    compression: snappy
  6. Start the MongoDB process again
  7. Join the member to the replica set (initial sync will happen)
  8. When the updated member is all caught up, pick another member and repeat the process.
  9. Continue until the primary is the only server left. At this point, you should step down the primary, but hold off switching storage engines until you are certain that the new storage engine meets your needs.
The Wrap Up

At this point I’ve explained how you can understand your options, where you can gain additional performance and what engines to evaluate. Please don’t forget to test your application with the new setup before launching into production. Please drop a comment below if you found this helpful or, on the other hand, if there’s something that would make it more useful to you. Chances are, if you’d find something helpful, the rest of the community will as well.

Categories: MySQL

Percona Live Featured Session with Bogdan Munteanu: Edgestore Multi-Tenancy and Isolation

MySQL Performance Blog - Wed, 2017-03-08 20:17

Welcome to another post in the series of Percona Live featured talk blogs! In these blogs, we’ll highlight some of the session speakers that will be at this year’s Percona Live conference. We’ll also discuss how these sessions can help you improve your database environment. Make sure to read to the end to get a special Percona Live 2017 registration bonus!

In this Percona Live featured session, we’ll meet Bogdan Munteanu, Software Engineer at Dropbox. His session is Edgestore Multi-tenancy & Isolation. Edgestore is Dropbox’s distributed metadata store, used by hundreds of products, services and features (both internal and external). Dropbox shares a single Edgestore deployment for all workloads, which has many benefits. At the same time it also poses challenges around multi-tenancy and isolation.

I had a chance to speak with Bogdan about Edgestore:

Percona: How did you get into database technology? What do you love about it?

Bogdan: I am very passionate about large-scale distributed systems, as well as storage in general. After joining Dropbox and learning about the scale, growth and technical challenges of Dropbox’s metadata store, I decided to jump in.

One thing I love about database and database services is that for every company and deployment, they are critical, highly impactful systems. You’re always in the thick of the action!

Categories: MySQL

How to Change MongoDB Storage Engines Without Downtime

MySQL Performance Blog - Wed, 2017-03-08 01:30

This blog is another in the series for the Percona Server for MongoDB 3.4 bundle release. Today’s blog post is about how to migrate between Percona Server for MongoDB storage engines without downtime.

Today, the default storage engine for MongoDB is WiredTiger. In previous versions (before 3.2), it was MMAPv1.

Percona Server for MongoDB features some additional storage engines, giving you the freedom for a DBA to choose the best storage based on application workload. Our storages engines are:

By design, each storage engine has its own algorithm and disk usage patterns. We simply stop and start Percona Server for MongoDB using different storage engines.

There are two common methods to change storage engines. One requires downtime, and the second doesn’t.

All the database operations are the same, even if it is using a different storage engine. From the database perspective, it doesn’t matter what storage engine gets used. The database layer asks the persistence API to save or retrieve data regardless.

For a single database instance, the best storage engine migration method is to start replication and add a secondary node with a different storage engine. Then stepdown() the primary, making the secondary the new primary (killing the old primary).

However, this isn’t always an option. In this case, create a backup and use the backup to restore the database.

In the following set of steps, we’ll explain how to migrate a replica set storage engine from WiredTiger to RocksDB without downtime. I’m assuming that the replica set is already configured and doesn’t have any replication lag.

Please follow the instructions below:

  1. Check replica set status and identify the primary and secondaries. (Part of the output has been hidden to make it easier to read.):
    foo:PRIMARY> rs.status() { "set" : "foo", "date" : ISODate("2017-02-18T18:47:54.349Z"), "myState" : 2, "term" : NumberLong(2), "syncingTo" : "adamo-percona:27019", "heartbeatIntervalMillis" : NumberLong(2000), "members" : [ { "_id" : 0, "name" : "test:27017", "stateStr" : "PRIMARY" (...) }, { "_id" : 1, "name" : "test:27018", "stateStr" : "SECONDARY" (...) }, { "_id" : 2, "name" : "test:27019", "stateStr" : "SECONDARY" (...) } { ... } ], "ok" : 1 }
  2. Choose the secondary for the new storage engine, and change its priority to 0:

    foo:PRIMARY> cfg = rs.config()
    We are going to work with test:27018 and test:27019. They are respectively the index 1 and 2 in the array members.
  3. Change the last secondary to the first instance to replace the storage engine:

    foo:PRIMARY> cfg.members[2].name test:27019 foo:PRIMARY> cfg.members[2].priority = 0 0 foo:PRIMARY> cfg.members[2].hidden = true true foo:PRIMARY> rs.reconfig(cfg) { "ok" : 1 }
  4. Check if the configuration is in place:
    foo:PRIMARY>rs.config() { "_id" : "foo", "version" : 4, "protocolVersion" : NumberLong(1), "members" : [ { "_id" : 0, "host" : "test:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "votes" : 1 }, { "_id" : 1, "host" : "test:27018", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "slaveDelay" : NumberLong(0), "votes" : 1 }, { "_id" : 2, "host" : "test:27019", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : true, <-- "priority" : 0, <-- "slaveDelay" : NumberLong(0), "votes" : 1 } ], "settings" : {...} }
  5. Then stop the desired secondary and wipe the database folder. As we are running the replica set in a testing box, I’m going to kill the process running on port 27019. If using services please run: sudo service mongod stop on the secondary box. Before starting the mongod service, add the --storageEngine parameter to the config file or application parameter:
    ps -ef | grep mongodb | grep 27019 kill < mongod pid>; rm -rf /data3/* ./mongod --dbpath data3 --logpath data3/log3.log --fork --port 27019 <strong>--storageEngine=rocksdb</strong> --replSet foo
    <config file> storage: engine: rocksdb
  6. This instance is now using the RocksDB storage engine and will perform an initial sync. When it finishes, to get the data from the primary node remove the hidden = false flag and let the application query this box:
    foo:PRIMARY> cfg = rs.config() foo:PRIMARY> cfg.members[2].hidden = false false foo:PRIMARY> rs.reconfig(cfg) { "ok" : 1 }
  7. Repeat step 6 for box test:27018, and use the following command as step 6. This makes one of the secondaries become the primary. Please be sure all secondaries are healthy before proceeding:
    foo:PRIMARY> cfg = rs.config() foo:PRIMARY> cfg.members[2].hidden = false false foo:PRIMARY> cfg.members[2].priority = 1 foo:PRIMARY> cfg.members[1].priority = 1
  8. When both secondaries are available for reading and in sync with the primary, we need to change the primary’s storage engine. To do so, please perform a stepdown() in the primary, making this instance secondary. An election is triggered (and may take a few seconds to complete):
    foo:PRIMARY> rs.stepDown() 2017-02-20T16:34:53.814-0300 E QUERY [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host '127.0.0.1:27019' : DB.prototype.runCommand@src/mongo/shell/db.js:135:1 DB.prototype.adminCommand@src/mongo/shell/db.js:153:16 rs.stepDown@src/mongo/shell/utils.js:1182:12 @(shell):1:1 2017-02-20T16:34:53.815-0300 I NETWORK [thread1] trying reconnect to 127.0.0.1:27019 (127.0.0.1) failed 2017-02-20T16:34:53.816-0300 I NETWORK [thread1] reconnect 127.0.0.1:27019 (127.0.0.1) ok foo:SECONDARY> rs.status()
  9. Please identify the new primary with rs.status() and repeat the step 5 and 7 with the old primary.

After this process, the instances will run RocksDB without experiencing downtime (just an election to change the primary).

Please feel free to ping us on Twitter @percona with any questions and suggestions for this blog post.

Categories: MySQL

Improving TokuDB Hot Backup Usage with the autotokubackup Command Line Tool

MySQL Performance Blog - Tue, 2017-03-07 00:44

In this blog post, we’ll look at how the command line tool autotokubackup can make TokuDB hot backups easier.

I would like to share an experimental tool named autotokubackup, for TokuBackup. This tool is aimed at helping simplify the life of TokuDB system administrators. autotokubackup is written in the Python language.

General information:

So why would you need this tool? Let’s clarify a bit what you might face while using tokubackup. You have a backup solution that you can use from the MySQL shell:

mysql > set tokudb_backup_dir='/var/lib/tokubackupdir';

Now you want to automate this process. The first problem is that the second backup will fail, because it’s required that the backup directory is empty before starting a backup process. One solution is to create time-stamped directories and for the backups.

Further, you have a backup policy that requires some other necessary files are copied as part of your backup process. You need to write a script to put those files into a separate folder under backup directory.

Another issue you will face is the lack of any clear output on backup progress. The shell just pauses until the backup completes. The one possible way to obtain information about the backup process is displaying the MySQL processlist in a separate MySQL shell. But it isn’t the best way, and there are some issues, as reported here: Unclear status information of backup state while taking backups using TokuBackup.

Generally, we need to know which files are backed up during the backup process. There should also be a clear message indicating the end of the backup process.

To make your life easier, the autotokubackup tool:

  • Automates the TokuDB database backup procedures
  • Creates timestamped backups inside the backup directory, overcoming the need for copy/remove old backups to empty the backup directory
  • Copies all necessary files for your backup policy (you can specify up to ten supplementary files to be in the backup directory as part of backup process)
  • Clearly describes what is going to be in the backup director, by showing newly created files inside backup directory
  • Clearly shows the end of backup process

To start, we only need two things:

  • Installed Percona Server with TokuDB engine + TokuBackup plugin
  • Installed Python3

To install the tool you can use following methods:

* From source:

cd /home git clone https://github.com/Percona-Lab/autotokubackup.git cd autotokubackup python3 setup.py install

* or via pip3:

pip3 install autotokubackup

The result will be something like:

Collecting autotokubackup Downloading autotokubackup-1.1-py3-none-any.whl Collecting watchdog>=0.8.3 (from autotokubackup) Downloading watchdog-0.8.3.tar.gz (83kB) 100% |████████████████████████████████| 92kB 8.2MB/s Collecting click>=3.3 (from autotokubackup) Downloading click-6.7-py2.py3-none-any.whl (71kB) 100% |████████████████████████████████| 71kB 10.6MB/s Collecting mysql-connector>=2.0.2 (from autotokubackup) Downloading mysql-connector-2.1.4.zip (355kB) 100% |████████████████████████████████| 358kB 4.7MB/s Collecting PyYAML>=3.10 (from watchdog>=0.8.3->autotokubackup) Downloading PyYAML-3.12.tar.gz (253kB) 100% |████████████████████████████████| 256kB 6.5MB/s Collecting argh>=0.24.1 (from watchdog>=0.8.3->autotokubackup) Downloading argh-0.26.2-py2.py3-none-any.whl Collecting pathtools>=0.1.1 (from watchdog>=0.8.3->autotokubackup) Downloading pathtools-0.1.2.tar.gz

After that there should be a configuration file for this tool located -> /etc/tokubackup.conf.

The structure of the config file is:

[MySQL] mysql=/usr/bin/mysql user=root password=12345 port=3306 socket=/var/run/mysqld/mysqld.sock host=localhost datadir=/var/lib/mysql [Backup] backupdir=/var/lib/tokubackupdir [Copy] # The following copy_file_x options allow you to copy various files together with your backup # Highly recommended; a copy of your my.cnf file (usually /etc/my.cnf) and any cnf files referenced from it (i.e. includedir etc.) # You can also include other files you would like to take a copy of, like for example a text report or the mysqld error log # copy_file_1= # copy_file_2= # copy_file_...= # copy_file_10= #copy_file_1=/etc/my.cnf #copy_file_2=/var/log/messages #copy_file_3= #copy_file_4= #copy_file_5= #copy_file_6= #copy_file_7= #copy_file_8= #copy_file_9= #copy_file_10=

You can change options to reflect your environment and start to use. Available command line options for the tool can be displayed using --help

tokubackup --help Usage: tokubackup [OPTIONS] Options: --backup Take full backup using TokuBackup. --version Version information. --defaults_file TEXT Read options from the given file --help Show this message and exit.

You can prepare different config files. For example, one for the slave. Specify using the –defaults_file option, and the overall result of the run should be something like the below:

tokubackup --backup --defaults_file=/etc/tokubackup_node2.conf Backup will be stored in /var/lib/tokubackupdir/2017-02-09_20-25-40 Running backup command => /home/sh/percona-server/5.7.17/bin/mysql -uroot --password=msandbox --host=localhost --socket=/tmp/mysql_sandbox20194.sock -e set tokudb_backup_dir='/var/lib/tokubackupdir/2017-02-09_20-25-40' mysql: [Warning] Using a password on the command line interface can be insecure. Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_data Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_logs Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_temp Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/log000000000006.tokulog29 Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.rollback Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.environment Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.directory Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tc.log Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/client-key.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/server-cert.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/server-key.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/ca.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/ca-key.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/auto.cnf Completed - OK

The backup directory will store the following:

ls -l 2017-02-09_20-25-40/ copied_files - Directory for copied files. global_variables - File for MySQL global variables. mysql_data_dir - Directory for copied MySQL datadir. session_variables - File for MySQL session variables. tokubackup_binlog_info - File for storing binary log position.(The new feature for TokuBackup) [Not released yet] tokubackup_slave_info - File for storing slave info.(The new feature for TokuBackup) [Not released yet]

That’s it. If you test it and find bugs, send a feature request to further improve our “helper.” Thanks!

Categories: MySQL

Webinar Thursday, March 9, 2017: Troubleshooting Issues with MySQL Character Sets

MySQL Performance Blog - Mon, 2017-03-06 23:52

Please join Percona’s Principal Technical Services Engineer, Sveta Smirnova as she presents “Troubleshooting Issues with MySQL Character Sets ” on March 9, 2017, at 11:00 am PST / 2:00 pm EST (UTC-8).

Register Now

Many MySQL novices find MySQL character sets support puzzling. But after you understand how it is designed, you will find it much more powerful than many other competing database solutions.

MySQL allows to specify a character set for every object, and change it online. For years this has helped to create fast applications that can work with audiences all around the world. However, it also requires any DBA troubleshooting character set issues to have a deep understanding of how they work. Different sort rules and collations can complicate the process.

In the webinar we will discuss:

  • Which character sets and collations MySQL supports
  • How they can be set
  • How to understand error messages
  • How to solve character sets/collations compatibility issues
  • What server, application, command line and graphical tool options are available
  • What to check first and how to continue troubleshooting
  • What the various compatibility issues are
  • How to convert data, created in earlier versions
  • What the best practices are

Register for the webinar here.

Sveta Smirnova, Principal Technical Services Engineer

Sveta joined Percona in 2015. Her main professional interests are problem-solving, working with tricky issues, bugs, finding patterns that can solve typical issues quicker and teaching others how to deal with MySQL issues, bugs and gotchas effectively. Before joining Percona, Sveta worked as Support Engineer in MySQL Bugs Analysis Support Group in MySQL AB-Sun-Oracle. She is the author of the book “MySQL Troubleshooting” and JSON UDF functions for MySQL.

 

Categories: MySQL

MySQL, –i-am-a-dummy!

MySQL Performance Blog - Mon, 2017-03-06 19:54

In this blog post, we’ll look at how “operator error” can cause serious problems (like the one we saw last week with AWS), and how to avoid them in MySQL using --i-am-a-dummy.

Recently, AWS had some serious downtime in their East region, which they explained as the consequence of a bad deployment. It seems like most of the Internet was affected in one way or another. Some on Twitter dubbed it “S3 Dependency Awareness Day.”

Since the outage, many companies (especially Amazon!) are reviewing their production access and deployment procedures. It would be a lie if I claimed I’ve never made a mistake in production. In fact, I would be afraid of working with someone who claims to have never made a mistake in a production environment.

Making a mistake or two is how you learn to have a full sense of fear when you start typing:

UPDATE t1 SET c1='x' ...

I think many of us have experienced forehead sweats and hand shaking in these cases – they save us from major mistakes!

The good news is that MySQL can help you with this. All you have to do is admit that you are human, and use the following command (you can also set this in your user directory .my.cnf):

mysql --i-am-a-dummy

Using this command (also known as safe-updates) sets the following SQL mode when logging into the server:

SET sql_safe_updates=1, sql_select_limit=1000, max_join_size=1000000;

The safe-updates and iam-a-dummy flags were introduced together in MySQL 3.23.11, and according to some sites from around the time of release, it’s “for users that once may have done a DELETE FROM table_name but forgot the WHERE clause.”

What this does is ensure you can’t perform an UPDATE or DELETE without a WHERE clause. This is great because it forces you to think through what you are doing. If you still want to update the whole table, you need to do something like WHERE ID > 0. Interestingly, safe-updates also blocks the use of WHERE 1, which means “where true” (or basically everything).

The other safety you get with this option is that SELECT is automatically limited to 1000 rows, and JOIN is limited to examining 1 million rows. You can override these latter limits with extra flags, such as:

--select_limit=500 --max_join_size=10000

I have added this to the .my.cnf on my own servers, and definitely use this with my clients.

Categories: MySQL

Writing Kindly

Xaprb, home of innotop - Sat, 2017-03-04 12:33

It makes me a bit uncomfortable when people say I’m a good person, because sometimes I’m actually a jerk! Thankfully, I’ve found that gentleness is a skill I can learn if I care enough to try, and I’ve gotten better at it over time. I’ve also found that when I’m nice and I focus on the positive, I get better results. Here are some things I’ve learned about how to be a kinder person in my writing.

Writing is hard, and an important part of my general life philosophy is “done is better than perfect.” In keeping with that, I’ve sometimes felt it was important or useful to publish something even if I thought it could still be improved. Sometimes, though, I’ve found that publishing things too soon can be hurtful.

Here is one example. In 2008, I was frustrated at MySQL’s marketing team, so I wrote a really hurtful post to send them a message. Of course, I did this without considering the advice of one of my coworkers, who always cautioned me to remember that people I disagree with are people with feelings, and they are getting out of bed every morning trying to do their best. I heard from some of those people and realized I’d been very wrong and had hurt them. This hurt me deeply too.

That was one extreme, a while ago. I got a little better, but I have occasionally done something similar again. For example, in 2015 I was frustrated at newcomers to Golang criticizing things they didn’t know deeply. I wrote a blog post telling them they were all wrong and ignorant. It came across as arrogant and judgmental, which it was.

Around the same time I started meditating, and actually noticing that I had feelings. (I had a tough childhood, and feelings weren’t allowed.) Things fell into place and I started to see the patterns. When I’m irritated, I act from a place of frustration, and I hurt myself and others. Awareness of this has helped me improve more rapidly.

But—still not perfect! One of the most popular posts I’ve written recently was about product-market fit. That post today is enormously different from what I first published. I wrote it because I kept trying to figure out what product/market fit was, and I had this sense that a lot of the advice I was reading and hearing didn’t quite strike the right chord for me. I got into a place of irritation and arrogance about my dawning understanding, and wrote an article that basically said everyone who’s talking about product-market fit is wrong about it. It wasn’t good at all and I didn’t like it.

Within hours of publishing it, with that uneasy irritation in the corner of my mind, I happened to read something from Julia Evans, whose writing never fails to teach and inspire me. I’m not sure what it was; perhaps it was this post or something similar. Julia’s writing often has a tone of here’s something I learned, isn’t learning awesome? And I realized that was not only the way I wanted to write the article, but also the way I wanted to choose to feel about my evolving understanding. Not exactly the same way Julia does—in a way that’s true to who I am—but nonetheless, using the principle that Julia demonstrates in her writing.

So I rewrote it, and then it became really popular. And as I reflected on why, I think it’s because it’s so much kinder now. Not only to my readers, whom I’m no longer insulting, but to myself, treating myself with compassion and celebration for the ongoing, never-complete process of learning, rather than superiority for the illusion that I’ve reached the state of knowing.

I’m pretty sure the Golang article would have been better if I’d written it in the same way. Here’s what I thought when I was new to Golang, and then I studied and it turns out there’s some depth to this stuff I didn’t see, and I know that makes X thing kind of annoying, but it’s for reasons. Would my criticism of the MySQL marketing team have been better if I’d written it differently? Perhaps. I know I wasn’t the only one who felt irritated, but you know what? Maybe some things just don’t need to be written at all.

Pic Credit

Categories: MySQL
Syndicate content