server monitoring Archives

What is Grafana? A look at its most important features for effective monitoring

What is Grafana, and what is Grafana monitoring? When Torkel Ödegaard took the decision in 2013 to fork Kibana, it heralded the beginning of the Grafana project, turning its forebear into a graph-focused and time-series dashboard tool. Ödegaard’s intention and guiding principle in creating it was to realize a dashboard that was dedicated to clarifying data by removing distractions and being a user-friendly and elegant solution. He seems to have succeeded in achieving his aims because half a million active installations don’t lie. These days you can find Grafana dashboards everywhere. Even if you’re unfamiliar with them, you may have seen one recently if you’re a fan of watching SpaceX launches!

Grafana Features

Grafana monitoring is achieved using panels. The basic building block for visualization in Grafana is the panel, and that panel can contain a graph, a Singlestat, a table, a heatmap, and freetext, and it can also integrate with both proprietary and community-created plugins too (like a clock or world map, for instance). Users are free to customize the style and format of each panel, and they can drag, drop, and resize them as they wish to create the ultimate visualization to suit their needs.

So, a Dashboard is a collection of panels, each of which holds a set of variables (things like sensor name, application, and server). These panels are arranged in a grid on the Grafana dashboard, and the user can change the data being scrutinized by switching variables, and that could be data from two different servers, for instance. Although the flexibility to customize views is one of the strongest Grafana features, users can just as easily pick up one of many ready-made dashboards to handle different data types and sources. Grafana’s large community of users and contributors has already created lots of them.

Grafana monitoring can include annotations to show particular events across the panels. Adding an annotation is achieved by putting in a custom request to Elasticsearch. Doing so makes this show up on the graph as a vertical red line. Hovering over an annotation then gives you an event description and tags (for instance) so you can track when the server returns a 5xx error code or when there’s a system restart, for instance. This makes it particularly easy to investigate system behavior and to track particular events and their consequences in an application.

Grafana Dashboard and Custom Web Apps

Grafana Dashboard and Graylog

Graylog can be used for the storage and management of web app logs and the monitoring of their performance, not just in production but also during the development stage. Grafana monitoring expresses these logs visually, to make analyzing the system more straightforward. You could legitimately describe Grafana as a web application load and performance user interface as well as a visitor flow tool. Graylog and Grafana work well together but there was no special effort made to integrate them. Graylog stores all log data in Elasticsearch, one of Grafana’s data sources, so it was easy enough to use one of the Elasticsearch indexes to connect Grafana to Graylog.

Visualizing Web Application Metrics in Grafana Dashboard

Grafana isn’t interested in error notifications or pure text logs because its main job is visualizing the data in tables, charts, and graphs. The developers created a custom Django module to track data on each web/worker request and processed response, not just reporting whether it succeeded or failed, but also providing a set of general and project-specific structured fields, including:

app version
ID for each unique request
response time and status
error code (if applicable)
IP address the request was sent from
user details (username for registered user’s email address, role, permissions)
device details

Django pushes these custom-structured analytical records into Graylog, which stores them in a different stream. Although Graylog dashboards can visualize this kind of data natively, they aren’t as adept at examining Grafana’s, so Grafana was adapted to visualize this analytical data. It can track application performance and load in real-time as well as retrospectively.

Top Features of Grafana

Grafana Labs is the name of the company that was created to push for the adoption of Grafana and to turn the project into a viable business. Whether you’re new to the Grafana dashboard or not, there’s a chance you may not be aware of the features that have been added to it both by Grafana Labs and its enthusiastic community.

Let’s take a look at some of the best ones:

Dashboard templating

This is one Grafana feature that’s really useful. It allows users to create a dashboard setup to suit their every need. And these templates don’t come with hardcoded values, which means that if you have a test server and a production server, the same dashboard will work with both. Templating lets you examine data at every level from the macro to the micro, so you can start with a whole country, for example, then drill down to a particular region, and keep going as far as granularity allows. These dashboards are then shareable with everyone from teams throughout your organization to the whole community.

Provisioning

It may be easy enough to set up a single dashboard with some clicking, dragging, and dropping, but some users need even more simplicity in a way that scales. So, Grafana features provisioning so you can automate setup using a script. Anything can be scripted in Grafana. For instance, when you want to create a new Kubernetes cluster, you can have Grafana automatically help with a script that already has the right server, IP address, and data sources set up and locked. This is also a way to control lots of dashboards.

Annotations

This Grafana feature lets you mark graphs, which is particularly helpful if you need to correlate data when something misbehaves. You can control-click and type on a graph to create your annotations manually, or data can be fetched from any source to populate them. (You can see an example of this in the way that Wikimedia uses annotations on its public Grafana dashboard.) A good use case would be automatically creating annotations at the time of releases. If you were to start seeing errors a little while after a new release, you could go back to your annotations and check if the errors correlate. This kind of automation is possible with the Grafana HTTP API. Lots of Grafana’s biggest customers use it for a wide range of tasks, with a common one being to set up databases and add users. This is an alternative to provisioning for automation, and there’s more you can do with it. For instance, DigitalOcean’s team used the API to include a snapshot feature that helps them to review dashboards.

Kiosk mode and playlists

Playlists are great for ‘rolling coverage’. You select which Grafana dashboards you would like to display on a monitor or TV, and it can cycle through them throughout the day. Kiosk mode lets you only show the user interface elements that you need in view-only mode. Useful tip: The Grafana Kiosk utility handles logins, switching to kiosk mode, and opening a playlist, so if a TV you want to use has no keyboard you can still set it up without hassle.

Custom plugins

You can extend Grafana’s functionality with plugins that offer extra tools, visualizations, and more. Popular examples include Worldmap Panel (which superimposes data on a map), Zabbix (which integrates with Zabbix metrics), and Influx Admin Panel (which enables database creation or lets you add users). These are just a couple of examples and there are many others besides them. Write a little code and Grafana can visualize anything that produces a timestamp. Also, Grafana Enterprise customers can access additional plugins that facilitate integrations with Datadog, New Relic, Splunk, and others.

Alerting and alert hooks

Grafana alerts can be sent through several different notifiers, including email, PagerDuty, or Slack or texts. If these don’t work for you, it’s easy enough to code alert hooks that create different notifiers.

Teams and permissions

Where an organization has one instance of Grafana and several teams, they usually like to have the option to enforce some dashboard segregation. It used to be the case that this wasn’t possible because Grafana automatically made everyone’s dashboards accessible to everyone else. The later edition of multi-tenant mode meant that users could switch organizations but couldn’t share dashboards. Some judicious hacks could enable both, so Grafana created an easier route to achieving this. It’s now possible to create a team of users and then assign permissions on folders, Grafana dashboards, and so on, right down to the data source level for Grafana Enterprise users.

SQL data sources

Grafana natively supports SQL, which helps you to graph any kind of data that might be held in an SQL database. High-end users are doing lots of interesting things with SQL data sources, including building business dashboards that “make sense to your boss’s boss,” (as one team put it).

Monitoring your monitoring

If you take your monitoring seriously enough to want to monitor your monitoring, Grafana features its own Prometheus HTTP endpoint that can be scraped by Prometheus, making it fairly easy to get statistics and dashboards. Once the enterprise version is up and running you’ll be able to get Google Analytics-style data access, so you can find out just how much CPU your Grafana is chewing through or how much time alerting takes.

Authentication

Grafana supports LDAP and OA and other authentication styles, and lets you map users to organizations. With Grafana Enterprise, it’s also possible to map users to teams: so if your organization uses its own authentication system, Grafana lets you map teams in your in-house systems to teams in Grafana, which automatically gives team members access to their own designated Grafana dashboards.

Grafana on Plesk

Plesk provides Grafana integration using Grafana Extension, the premier open-source software for time-series analytics. Grafana can turn all kinds of data into all kinds of visually appealing graphs and dashboards that can be customized in endless ways.

You can use Grafana to:

visualize data pulled from default sources, which the Plesk team can integrate for you. The default source at present is the Advanced Monitoring extension, which gathers metrics on server health.

as a more experienced Grafana user, you can pull data from any source that will integrate with Grafana. If you’d like to do this, just give the appropriate permission to the Grafana administrator.

Grafana server monitoring

Server Management – What Is It All About?

Various aspects of server management - Plesk

Nowadays it’s really easy to set up a server, with many essential server management functions you need via auto-install. Servicing your server is a different matter. Because keeping tabs on maintenance plus performance and utilization levels can be time-consuming. It gets even more difficult to perform server administration when your operations include multiple servers. Tapping into a single vendor can trap you, consider using a mix of vendors for your server operating systems.

Manual management across big installations is time-consuming. You just can’t manually log into server administration consoles on a machine by machine basis. Server management software are therefore crucial to successful management procedure. Most of these tools come with remote administration and machine monitoring. Thus, enabling you to manage machines across a range of sites in an efficient manner.

Server Types

There are application servers, web servers, email infrastructure servers, and file storage servers – classified as network attached storage. You may also use the term “server” to identify the hardware part under your application layer. Your management tasks and the performance evaluation will be different depending on the each separate case.

Cloud services have completely changed server monitoring and administration routine. If you use cloud services – the applications, e-mail and storage facilities you use will be offsite. Indeed, keeping track of all these co-located servers can become a real server management nightmare.

Server Monitoring

The first step in infrastructure management is getting a single interface to monitor all your servers from your single workstation. It’s important to get real-time data on your hardware. Your server checking software should show you instant facts covering processor use, memory utilization, and disk space availability. On top of that, you must be able to see which processes are operating on the server. Including what amount of resources each of these processes consume.

Essentially you need to rely on a tool which can provide alerts based on live data evaluation. On top of collecting data, your checking tool must be able to automatically flag issues to someone who can respond to these issues. This could include notification via e-mail or SMS.

Finally, we would highlight that it is key to keep on top of utilization. If you see your apps are increasingly taxing your machine, you can add capacity before a demand spike causes issues.

Server Management

Planning the capacity of your server is a new area of server management. You never want to provide an excess of machine capacity. Doing so can mean wasting hardware resources, paying too much for utilities and spending too much on support. However, you do need to plan for surges in demand. So a bit of spare resources is always a good idea.

Note that provisioning for computing requirements involve also provisioning other areas such as your physical network. All while providing physical space for equipment including power supply is also important. You also need to account for staff requirements. These are variables that create a unique situation for your business. Meaning you need to pick a server management tool that fits your company the best.

Server management software should make it easy for sysadmins to keep an eye on hardware availability. Alongside the expiration of software licenses, patches that are due and automated alerts when unauthorized software installations are performed.

Server Administration Roles

Senior staff is generally responsible for infrastructure management. Meanwhile, day-to-day maintenance and checking can be allocated to more junior staff. Or be fully-automated using management software.

Your company size of number of admin employees defines your flexibility regarding user roles and restricting access to system data. Small businesses, for example, may have just a single person in charge of machines. With a single associated user role and a single user account for your server management tool.

On the flip side, larger groups can benefit from allocating specialized tasks to individual employees. Hence making managing access to management functions a priority. To effectively manage your infrastructure, you need to use a range of software, including remote server administration tools.

Choosing Server Management Tools

You’re likely to continue using the management tool you choose for a long period of time. So it’s worth carefully considering your choice. Factors you should keep in mind include:

Vendor Compatibility

Yes, you may be satisfied with your current hardware provider. But as you upgrade and renew your machines you may change providers or introduce a mix of providers. Don’t choose a server monitoring tool which restricts you to a specific provider. Even if all your equipment is from a single vendor, try to ensure long-term flexibility via a tool compatible with multiple vendors.

Server Monitoring Overheads

Every piece of software consumes resources, including server management and checking software. In most cases, a tool is likely to fit clients with operations that are a certain size. Importantly, you should avoid getting a tool which slows down your operations. Or which generates too much traffic on your network. Not sure what the resource impact of a tool will be? Many vendors offer trial periods on their software – just try it out first.

Match Your Server Administration Requirements

Different software come with different features, so try to match the tool you buy to your company’s oversight needs. Thought an extremely comprehensive package may look attractive, you should be careful not to buy software that you will never use. For small environments, buy a simple tool. For larger environments, consider a tool which can deal with complexity and which includes group-management functionality. However, never buy a tool that’s so complicated you end up not using it.

Server Management Roles

It’s useful to be able to grant restricted access to management consoles. Even if there’s just a single sysadmin looking after servers. You can, for example, give management staff the ability to directly view reports so that they can draw their own conclusions. Or, you may in future employ an assistant in which case you would need your junior employee to access functionality without giving full control to your assistant.

Scaling Server Monitoring

Your computing requirements may change in future. Smaller outfits should consider buying a cut-down edition of a tool which is made for large operations. Should you need to upgrade you can simply step up inside the product family, so you don’t need to retrain. Packages which only work with smaller environments may mean that you will need to switch vendors later on which can involve a learning curve.

Automating Server Management Processes

With the complex server environments so common nowadays, the mere ability to perform checking is no longer enough. Instead, you need to be able to automate the regular server administration tasks that are time-consuming. Good software can reduce much of the server administration tasks down to simply checking logs. It’s a job that gives the opportunity for interns to gain knowledge while freeing up the time of expensive sysadmins.

Server Monitoring Options

No question that overseeing your infrastructure’s important, but you don’t need to spend lots to get good insight into your servers. One way to obtain good value is by combining software, including application and network control.

At a basic level, software will simply show you the state of your servers. Including facts such as processor use and disk space utilization. These are important facts, but they do not always give much insight into the actual user experience. Note that just focusing on machine statistics may mean that you don’t see other important factors, including network performance or the uptime of your cloud service providers.

In essence, performance oversight is a large remit and it is worth using a complete system which can take care of all your IT requirements. So that you have a better idea of all the tweaks you need to perform to ensure your infrastructure is in good shape.

You’ll be doing a lot of reading as you search for a machine monitoring package. But you won’t really understand a server monitoring system without giving it a try – first-hand. Thankfully most server management/monitoring software let you try for free. As a result, you can make a much more accurate assessment regarding whether a particular tool is a good fit for you.

Yes, the evaluation process can take time since you’ll be spending time setting up a server management tool. Even if you have to perform this task several times, you need to get to the end-goal of finding the right fit.

Keeping Tabs with Plesk Obsidian’s Advanced Monitoring Extension [VIDEO]

Plesk is continually evolving. We released Plesk Obsidian just a few months ago, and it’s by far our most powerful management platform yet. Jam-packed with new features, Plesk Obsidian also introduces six new extensions that make the Plesk experience better than ever. Each extension offers users enhanced usability, tougher security, and maximum productivity. One of these new extensions is the Advanced Monitoring extension.

Advanced Monitoring replaces the Server Health Monitor component as the main feed for monitoring the health of your server. The Advanced Monitoring extension works in tandem with Grafana – an extension that displays server metrics as visuals, graphs and dashboards. The result is a powerful combination that optimizes both the flexibility and control of your Plesk account.

This article will fill you in on everything you need to know about the new Advanced Monitoring extension. We’ll first look at why the extension is so important and the problems it solves. Then, its main features, and finally how you can start using it.

What problems does Advanced Monitoring solve?

Keeping track of your servers can be a tricky and time-consuming task. But anyone who’s suffered an unexpected cut knows it can ultimately save you a whole lot of time and hassle. That’s why, a tool that can automate the monitoring of your servers is an invaluable asset. However, it’s important that it actually makes the process easier and upholds server monitoring best practices.

First on the best practice list is that the tool can watch various system metrics around the clock. Alerting you whenever there’s a problem. The Advanced Monitoring extension does this by tracking your server resources usage day and night.

It then notifies you when the usage of one or more resources reaches a pre-defined threshold. Because it’s essential to be instantly alerted of critical issues in order to handle them right away. Advanced Monitoring sends you emails according to your preferences and limits so you never have to worry about breaks again.

Another best practice is being able to monitor current and historical data of your server usage. This can help you get a feel for the overall state of your Plesk. Ensuring you don’t overload your server, and allowing you to plan accordingly for future needs.

Of course, all this data is useless unless it’s offered in a clear, easily-digestible manner. Grafana is the leading open source software for time series analytics. The extension displays data in Advanced Monitoring through graphs and dashboards that are highly customizable and visually appealing.

Main features of Advanced Monitoring

Let’s take an overview of Advanced Monitoring’s full feature list. With the Advanced Monitoring extension, you can:

See detailed reports on your server health
See how server health parameters change with time
Change the displayed time period
Keep track of the system resources’ usage on your server
Configure email notifications to inform when one or more resources’ usage reaches a pre-defined threshold (this feature is on its way).

One of the best things about the extension is that it’s ready to go out-of-the-box. Both the Advanced Monitoring and Grafana extensions are installed in Plesk Obsidian by default. You can find them located in the main left panel. The resources and services monitored by Advanced Monitoring are separated into five categories, with each one shown on its own tab.

If you upgrade to Plesk Obsidian from Plesk Onyx, Advanced Monitoring will automatically install and replace Server Health Monitor.

How to use the Advanced Monitoring extension

It’s incredibly easy to monitor your resources and server health using Advanced Monitoring. Most of its features are ready to go, but you can also customize it to suit your needs.

For example, you can select a time period for which Advanced Monitoring generates graphs. Or specify a custom period by selecting “Custom time range”. This can help you find out how your parameters change over time. As well as identify the periods when your resources usage is maximal or minimal.

You can also track your server health. Not only by looking at resources graphs, but by setting up and receiving Advanced Monitoring notifications. Advanced Monitoring can show notifications in Plesk’s interface when your resource usage exceeds its threshold and/or returns to normal value. The notifications only show for resources with notifications turned on and which have thresholds set in the Advanced Monitoring settings.

Certain resources already have their notifications turned on and their thresholds set by default. When a default threshold’s exceeded, it means a parameter is close to its critical value, which can be problematic. The automated-add on Grafana allows you to see detailed, visual reports on server health. And follow the real-time evolution for parameters that are of interest to you.

It offers numerous types of dashboards and can be used in two main ways:

With the data collected from default sources which were integrated for you by us.
(If you’re an experienced Grafana user) By harvesting data from any source you can integrate Grafana with.

We’re very excited about all Plesk Obsidian’s powerful new extensions. If you have any questions about Advanced Monitoring, be sure to leave us a comment below. Or get in touch with our friendly team of Pleskperts.