How to upgrade Splunk Enterprise
This article is to share my experience regarding splunk enterprise upgrade on Linux Servers . To be honest ,Splunk website offers many articles and guides on how to upgrade Splunk core system . However one as to read many articles and follow links within links which can be confusing at times. In my article , I have tried to simplify it .
At the bottom of this article , I will be posting all the links that I used during this activity. I am also assuming that you are a splunk administrator who is familiar with his Splunk environment and know if it is an All in one Setup or distributed/Clustered Setup .
For most large organizations , you will have a distributed OR clustered Setup .
Normally a Distributed Clustered Setup has following components
Deployer /DMC
Deployment Server
License Server/Cluster Master
Search Head
Heavy Forwarder
Indexer
Hint : You can all your Splunk Instances on your DMC (Distributed Management Console) . Go to DMC | Monitoring Console | Instances .
Tip : If your environment is Distributed/ Clustered than you need to start Upgrade Process with less critical Servers and move your way up to most critical Servers like Heavy Forwarders , Search Head and Indexers .
In many cases, you upgrade the Splunk software by installing the latest Splunk Enterprise package over your existing installation. Similarly when you upgrade on Windows systems, the installer package detects the version that you have previously installed and offers to upgrade it for you.
Kindly ensure that you are able to run the commands given in this article before you actually go for an Upgrade . It might be that you have a limited user who is unable to run the backup OR Service start/stop commands .So Here are some Key Steps for an Upgrade on Linux Servers which are mentioned below
Download the required files
Copy the files to Splunk Server
Check KV Store Status ( if applicable)
Backup KV Store (if applicable)
Backup existing Splunk Server
Stop Splunk Service
Run the Upgrade
Start the Service and Accept the Licence
Lets go through each step one by one .
Download the required files
First thing that you need to do do is to download the required files . In my case , I was upgrading from 7.3.3 and my target release was 9.0.1 so I had to download two different versions i.e 8.1.6 and 9.0.1 . Also I had to upgrade in two phases .
Upgrade Splunk Enterprise Core from 7.3.3 to 8.1.6
Upgrade Splunk Enterprise Core from 8.1.6 to 9.0.1
I was able to download all the files from Splunk Website easily .
Copy the files to Splunk Server
Once files were downloaded , I used winscp software to copy them to /home/splunk directory on Splunk Server . You can use winscp or any other Software for this .
Check KV Store Status
In Splunk different Servers have different roles and some Servers are running with Kvstore . You can see this information from your DMC Instances Tab . So Before starting the Process , for Some Servers who were using KV Stores , I ran this command to check the status
/opt/splunk/bin/splunk show kvstore-status
Backup KV Store
Depending on the output , If kvstore was running fine ,I backed up the configuration using this command .
/opt/splunk/bin/splunk backup kvstore -archiveName kvstorebkup
For some servers ,I had to use sudo as my user was restricted .
sudo /opt/splunk/bin/splunk backup kvstore -archiveName kvstorebkup
Please note that this is just a precaution as the Upgrade Process does not mess up with Kvstore and does not cause any trouble .
Backup existing Splunk Server
You can backup the entire installation or you can backup only the etc directory which has all the necessary and required configurations .
In my cases ,some servers I backed up entire directory . However once I got the hang of it , I only backup etc directory using the command below .
In my setup , Splunk was installed under /opt directory so my path was /opt/splunk .
First make a directory : mkdir backup
Now issue command as written below .
tar -zcvf /home/splunk/backup/splunk-config-backup.tar.gz /opt/splunk/etc/*
If you do not have full rights than better to use sudo command .
sudo tar -zcvf /home/splunk/backup/splunk-backup.tar.gz /opt/splunk/etc/*
At the same time ,better to check Splunk Version .
/opt/splunk/bin/splunk version OR sudo /opt/splunk/bin/splunk version
Stop Splunk Server
Once the configuration is backed up . we are ready now . Lets stop the Service .
/opt/splunk/bin/splunk stop OR sudo /opt/splunk/bin/splunk stop
Run the Upgrade
Doing the Upgrade on Linux is actually unzipping the files over existing Installation directory to replace files . That’s why the Service should be stopped .
Assuming that the setup file was copied in /home/splunk path ,use this command .
tar zxf /home/splunk/splunk-8.1.6-c1a0dd183ee5-Linux-x86_64.tgz -C /opt OR
sudo tar zxf /home/splunk/splunk-8.1.6-c1a0dd183ee5-Linux-x86_64.tgz -C /opt
Please note that while Upgrading Search Heads and Indexers , you should enable Cluster Maintenance mode on your Cluster Master
/opt/splunk/bin/splunk enable maintenance-mode
Start the Service and Accept the Licence
Once all the files are replaced , use this command to start the service . Please accept the license along with the command else you will have to go through an 8 page license agreement file .
/opt/splunk/bin/splunk start — accept-license — answer-yes OR
sudo /opt/splunk/bin/splunk start — accept-license — answer-yes
Once Splunk Service is started ,you can run this command to see the Updated Version .
/opt/splunk/bin/splunk version OR sudo /opt/splunk/bin/splunk version
Alternatively you can also Login to splunk and check .
Best Practices
Overall the Process looks very simple however one should always be prepared for the worst so follow these best practices .There is an amazing Upgrade sheet called General Process to upgrade Splunk Enterprise which is very helpful and can be used as a reference guide.
- Identify your Architecture. list out all Splunk Components and App Versions and check if they are compatible with the target version or not .
- Start backing up Configurations . Go to your DMC ( Distributed Mgmt Console) and take snapshots for all Splunk Components . It will help you set a benchmark and you can compare after Upgrade .
- Start Upgrade with less critical servers like deployment server or deployer . Keep indexer as the last one .
- Verify each Server after upgrade to see if it is working as expected
References
https://community.splunk.com/t5/Installation/What-s-the-order-of-operations-for-upgrading-Splunk-Enterprise/m-p/408003
https://docs.splunk.com/Documentation/Splunk/7.3.3/Installation/HowtoupgradeSplunk
https://docs.splunk.com/Documentation/Splunk/7.3.3/Admin/BackupKVstore
https://docs.splunk.com/Documentation/Splunk/8.1.6/Installation/UpgradeonUNIX
https://docs.splunk.com/Documentation/ES/7.0.0/Install/Upgradetonewerversion#Version-specific_upgrade_notes
https://docs.splunk.com/Documentation/ES/7.0.0/Install/DeploymentPlanning#Splunk_Enterprise_system_requirements
https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/CompatMatrix
